外观
论文arXiv 2610.1017821 天前
视觉-语言-动作模型理解指令吗？关于语言 grounding 的机制可解释性研究
原文链接：Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding（arXiv 2610.10178 · 发布日期：2026-10-07）
Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding
研究VLA动作生成是否依赖语言指令，用激活与归因 patching 分析两模型，并在 LIBERO 样本上验证。
核心要点
- 问题在于VLA缺少显式 grounding 模块，可能依赖视觉线索和表面相关而非指令。
- 方法是在LIBERO样本中以五种策略破坏任务指令，并对动作生成模块残差流做 patching。
- 实验发现两模型对抽象改写和不存在物体较不敏感，但对空指令和方向语言反应强。
- 敏感性位置因模型而异；归因 patching 可靠性也呈模型依赖，GR00T N1.7更一致。
验证范围与局限：验证范围限于LIBERO基准输入样本上的π0.5与GR00T N1.7两个模型。
英文摘要 (Abstract)
Vision-Language-Action models are designed to generalise across environments and task descriptions, raising the question of whether their action generation actually depends on the language instruction, or whether they largely rely on visual cues and superficial correlations. Robustness to variance in the visual and linguistic observation space is critical for real-world deployment, yet VLAs lack explicit grounding modules and instead rely on the intrinsic language grounding capabilities of their Vision-Language model backbones. For this reason, we conduct a controlled mechanistic interpretability study on the language grounding capabilities of two state-of-the-art Vision-Language-Action models, $π_{0.5}$ and GR00T N1.7, by applying activation and attribution patching to the residual stream of the action generation modules. We systematically corrupt the task instruction of input samples of the LIBERO benchmark following five strategies: synonym replacement, semantic scaling, directional corruption, random object substitution, and empty string. Our experiments find that both models are comparatively insensitive to abstract rephrasing and to referencing non-existent objects, but react strongly to empty task descriptions and, especially, to directional language. During action generation, this sensitivity is concentrated in different loci for each model: mainly in the early, periodic cross-attention layers for GR00T N1.7, versus distributed across the earliest and selected later layers for $π_{0.5}$. For GR00T N1.7, directional perturbations drive some of the largest causal effects while leaving the internal representational geometry comparatively unchanged, a dissociation we do not observe clearly for $π_{0.5}$. Finally, the reliability of attribution patching is model-dependent: it closely tracks activation patching for GR00T N1.7 but not for $π_{0.5}$.
作者：Theodor Wulff, Angelo Cangelosi
来源
本页事实对应的公开页面，以原文为准。