外观
论文arXiv 2610.1663416 天前
Language2Drive：Vision-Language-Action 模型在自动驾驶中真的遵循指令吗？
原文链接：Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?（arXiv 2610.16634 · 发布日期：2026-10-12）
Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?
提出Language2Drive及PCU，在CARLA仿真、重建真实场景和开环基准中检验自动驾驶VLA的指令跟随能力。
核心要点
- 现有闭环基准偏重路线完成、安全和规则遵守，难判断语言是否决定可行驾驶行为。
- PCU固定初始驾驶场景、仅改变指令，用于区分孤立指令成功与反事实成对成功。
- 在CARLA仿真、重建真实场景和开环基准评估代表性驾驶VLA，显示驾驶能力强但语言控制弱。
- 作者发布CARLA-Language2Drive仿真器；结果指出常规驾驶分数会高估语言跟随能力。
验证范围与局限：验证范围为闭环CARLA仿真、重建真实世界场景和开环驾驶基准。
英文摘要 (Abstract)
Recent Vision-Language-Action (VLA) models increasingly use natural-language instructions to condition autonomous driving behavior. Yet existing closed-loop benchmarks primarily measure route completion, safety, and rule compliance, leaving a fundamental question unresolved: does language actually determine which feasible behavior the vehicle executes? A trajectory may be safe and successful while still violating or even ignoring the passenger's instruction. To address this, we introduce \ltd, a benchmark designed to directly test this capability. Its key evaluation unit, the paired counterfactual unit (PCU), holds the initial driving scene fixed while changing only the instruction. This design enables us to distinguish isolated instruction success from counterfactual paired success across four instruction-following scopes. To test whether this distinction persists across evaluation regimes, we instantiate Language2Drive in complementary closed-loop CARLA simulation, reconstructed real-world scenes, and open-loop driving benchmarks, including our released CARLA-Language2Drive simulator with purpose-built maps. Evaluating representative driving VLAs across these settings reveals a striking mismatch between strong driving competence and weak language control: 93.2% of successful rollouts do not have a successful counterpart under the alternative feasible instruction. This failure is dominated by one-sided completion, where a model repeatedly executes one preferred behavior but fails when the same scene requires the counterfactual alternative. These results expose two fundamental limitations of current evaluation: isolated success and driving scores, which are widely used by conventional driving benchmarks, can substantially overstate language-following capability, while current driving VLAs themselves lack reliable instruction following under matched counterfactual tests.
作者：Zihan You, Zehua Fan, Wenqi Lyu, Aoqi Wang, Ni Wang, Yifei Yang, Linge Zhao, Huan Li, Sining Ang, Yuguang Yang
来源
本页事实对应的公开页面，以原文为准。