外观
论文HF Daily Papers · arXiv 2610.278618 天前
RoboCasa-Shift基准发布：VLA策略长程任务鲁棒性阶段感知评测
本文由 AI 根据公开内容整理，可能有误，请以原文为准：RoboCasa-Shift: Stage-Aware Robustness Evaluation of Vision-Language-Action Policies in Long-Horizon Household Manipulation（HF Daily Papers · arXiv 2610.27861）
核心内容
为解决视觉语言动作（VLA）模型鲁棒性评估中聚合成功率掩盖长程任务失败模式的问题，研究团队发布RoboCasa-Shift诊断基准。该基准基于RoboCasa365构建，采用受控单因素扰动与配对干净/扰动轨迹设计，对GR00T N1.5策略在4项长程家庭操作任务中进行了1040次正式评测。实验覆盖4种扰动因子及3级校准严重度，发现鲁棒性响应在不同任务间差异显著，且性能随扰动增加并非总是单调下降。
技术方案核心在于阶段感知诊断机制，通过匹配底层设置仅改变目标扰动的成对rollouts来隔离变量。数据显示，Camera L2与Gaussian Blur L1虽导致相同的端到端成功率下降（ΔSR=0.188），但首阶段失败率分别为19.0%和36.4%，揭示了故障演进路径的本质差异。此外，特定相机条件下8/80个匹配片段结果改变但整体ΔSR为0，证明单一指标失效。该工具支持联合捕获结果退化、配对敏感性及故障进程，代码已在GitHub开源。
关键看点
- 基准架构：基于RoboCasa365构建，采用配对干净/扰动rollout设计，确保每对轨迹仅存在单一目标扰动差异，支持阶段感知诊断分析
- 评测规模：针对GR00T N1.5策略完成1040次正式测试，覆盖4项长程任务、4种扰动因子及3级严重度，包含导致完全失败的极端条件
- 非单调退化：实验发现性能下降不随扰动严重度单调变化，如某相机条件下8/80匹配片段结果翻转，但整体成功率变化ΔSR仍为0
- 阶段异质性：Camera L2与高斯模糊L1虽致相同ΔSR=0.188，但首阶段失败率分别为19.0%和36.4%，证实端到端指标无法反映故障深度
工程与产业观察
该工作将VLA鲁棒性评估从单一端到端成功率推向多维执行属性分析，对具身智能算法迭代具有关键参考价值。其揭示的“同损异因”现象表明，仅优化最终指标可能掩盖早期感知或规划缺陷。工程落地中应引入阶段感知诊断协议，结合配对敏感性测试定位长程任务中的脆弱环节，避免模型在分布偏移下出现不可解释的系统性崩溃。
英文摘要 (Abstract)
Vision-language-action (VLA) robustness is commonly summarized by aggregate task success, which obscures how individual executions change under distribution shifts and where failures emerge in long-horizon tasks. We introduce RoboCasa-Shift, a diagnostic robustness benchmark built on RoboCasa365 that combines controlled single-factor shifts, matched clean-perturbed rollouts in which each pair shares the same underlying settings and differs only in the target shift, and simulator-grounded stage-aware diagnostics. We evaluate GR00T N1.5 on four long-horizon tasks that pass a predefined clean-performance gate under four perturbation factors and three calibrated severity levels, including conditions that drive the evaluated policy to complete task failure, totaling 1,040 formal rollouts. Robustness responses vary substantially across factors and tasks, and increasing severity does not always produce monotonic degradation. For example, one camera condition yields ΔSR=0 despite 8 of 80 matched episodes changing outcome. Stage-aware analysis further shows that the same end-to-end degradation can correspond to different execution-depth profiles. Camera L_2 and Gaussian blur L_1 both yield ΔSR=0.188, yet their first-stage failure rates are 19.0% and 36.4%, respectively. Our findings suggest that VLA robustness should be viewed as a multi-dimensional property of execution rather than a single end-to-end performance measure, motivating evaluation protocols that jointly capture outcome degradation, paired outcome sensitivity, and failure progression. The code is publicly available at https://github.com/HaochengWang23/RoboCasa-Shift.
Hugging Face Daily Papers 热度：2 upvotes 作者：Haocheng Wang, Youquan Liu, Weiyu Guo, Aokai Zhang, Donghang Wu, Hongyu Liu, Sha Zhang, Weidong Yang, Ben Fei
来源
本页事实对应的公开页面，以原文为准。