Abstract

Imitation Learning (IL) is widely used for autonomous driving but remains vulnerable to shortcut learning, where policies exploit spurious correlations rather than causal driving cues, undermining generalization and robustness under distribution shift and OOD conditions. We present CAR Planner, which augments the attention mechanism with a constrained formulation: an inequality constraint limits the mean deviation of ego-state cross-attention weights from uniform, enforced via the Augmented Lagrangian Method (ALM). This regularizer curbs over-reliance on a few channels and encourages balanced state representations, adding negligible training overhead and no inference cost. On nuPlan, CAR Planner shows markedly smaller performance drops when ego-state channels are reduced, providing strong evidence of robustness against shortcut reliance. Furthermore, it consistently outperforms both an unconstrained baseline(without constraint) and state-of-the-art dropout-based methods in challenging scenarios, while also producing smoother and more stable driving. These results indicate that CAR Planner enables more robust IL for autonomous driving.