论文HF Daily Papers · arXiv 2611.251794 天前
SimAudit审计仿真操作基准:检出61处成功判定器缺陷
本文由 AI 根据公开内容整理,可能有误,请以原文为准:Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks(HF Daily Papers · arXiv 2611.25179)
核心内容
针对仿真操作基准依赖手写成功判定器(Success Checkers)导致的评估偏差,研究团队提出基于LLM的自动化审计管线SimAudit。该工作系统性地检测了判定器代码与任务指令的一致性,在RoboTwin 2.0、RoboCasa365及RoboDojo三个主流基准的183个测试案例中,经视觉验证与去重后确认61处独立缺陷,其中包含40次“幻影成功”(Phantom Success)和21次“静默失败”(Silent Failure),量化揭示了当前仿真评测体系的系统性漏洞。
SimAudit通过分析任务指令与判定器代码识别潜在分歧场景,并在仿真中重现这些边缘案例,利用多视角渲染验证实际行为是否满足指令。审计发现RoboCasa365主要存在接受未完成任务的幻影成功,RoboDojo倾向于拒绝有效执行的静默失败,而RoboTwin 2.0两类错误并存。结果表明,仅通过专家演示验证判定器不足以覆盖学习策略的多样性,必须引入针对不完整执行与有效替代方案的对抗性测试协议。
关键看点
- 审计管线:SimAudit利用LLM分析指令与代码分歧,结合仿真重现与多视角视觉验证,自动化检出判定器逻辑缺陷
- 缺陷分布:在三大基准中确认61处缺陷,含40次幻影成功与21次静默失败,覆盖49个具体任务场景
- 基准差异:RoboCasa365多发接受不完整执行的幻影成功,RoboDojo多为拒绝有效操作的静默失败,RoboTwin 2.0两类错误共存
- 评测启示:证实仅靠专家演示无法保证判定器鲁棒性,需建立涵盖非标准完成路径与部分进度的对抗性验证协议
工程与产业观察
该研究直击具身智能仿真训练中“奖励欺骗”与“假阴性”痛点,证明手写规则难以适配策略探索的多样性。对于算法工程师,这意味着在Sim-to-Real迁移前必须对Benchmark进行自动化审计;对于基准构建者,应摒弃单一专家演示验证,转向包含边缘案例的压力测试,以确保训练信号的真实有效性。
英文摘要 (Abstract)
Simulated manipulation benchmarks measure policy performance using hand-written success checkers that label each episode as a success or failure. Passing an expert demonstration shows that a checker accepts one way of completing the task. Learned policies, however, may stop after partial progress or complete the task in a different way. A checker may then accept an incomplete execution (phantom success) or reject an execution that satisfies the instruction (silent failure). In this paper, we introduce SimAudit, an LLM-based pipeline that audits success checkers against task instructions. SimAudit analyzes the instruction and checker code to identify scenarios where the two may disagree, recreates those scenarios in simulation, and runs the original checker. Multi-view renderings are then used to verify whether the observed behavior satisfies the instruction. Across RoboTwin 2.0, RoboCasa365, and RoboDojo, 183 simulated test cases produced the predicted checker verdict. Visual verification and deduplication yielded 61 distinct defects across 49 tasks, including 40 phantom successes and 21 silent failures. Most defects in RoboCasa365 accepted incomplete executions, most in RoboDojo rejected valid executions, and both error types were common in RoboTwin 2.0. These findings suggest that validating success checkers requires testing both incomplete executions and valid alternatives to expert demonstrations, and motivate more rigorous evaluation protocols for future benchmarks.
Hugging Face Daily Papers 热度:1 upvotes 作者:Zirui Song, Jiadong Zhang, Zheng Lu, Xiaosong Ma, Xiaojun Chang, Xiuying Chen
外部来源
本页事实对应的公开页面,以原文为准。(点击将离开本站)