外观
论文arXiv 2610.2337212 天前
超越成功率：视觉-语言-动作系统中记忆的干预式评估
原文链接：Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems（arXiv 2610.23372 · 发布日期：2026-10-16）
Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems
提出 IMEP 在 VLA 系统中干预评估记忆作用，并在多类记忆与 ChainBench 上验证其影响。
核心要点
- 仅做成功率消融会同时改变信息和计算，难以证明记忆中的任务相关事实被使用。
- IMEP 在检索接口比较目标事实替换、事实保持替换、微小数值扰动和精确副本。
- 跨回合 learned-bank 实验中，ULP 干预使 11.9% 的种子配对结果发生变化。
- ChainBench 要求使用当前观测缺失的历史信息，完整记忆将成功率从 39.1% 提升到 74.3%。
验证范围与局限：验证涵盖回合内、跨回合与 agentic memory，摘要未说明仿真或真机平台。
英文摘要 (Abstract)
When we say that a robot uses memory, does that memory actually play a role? Success-rate ablations alone cannot establish the use of task-relevant facts: they change information and computation together, while aggregate rates can hide offsetting gains and losses. We propose an Interventional Memory Evaluation Protocol (IMEP) for vision-language-action (VLA) systems to address four questions: (1) Does memory provide a net benefit? (2) Under what conditions does memory help? (3) Do changes to memory alter task outcomes? (4) Are these effects specific to task-relevant facts? At the retrieval interface, we compare target-fact substitutions with fact-preserving replacements, minimal numerical perturbations, and exact replicas, while assessing decision correctness separately from motor execution success. We evaluate within-episode, cross-episode, and agentic memory. In cross-episode learned-bank experiments, a one-unit-in-the-last-place (ULP) intervention, including its installation procedure, changes 11.9% of seed-paired outcomes, while changing key facts produces no clear additional effect over fact-preserving replacements. To examine when memory helps, we introduce ChainBench, where correct decisions require historical information unavailable in current observations. The full memory configuration raises task success from 39.1% to 74.3%, matching the oracle-decision condition under shared recorded executions. These results show that correct history can support robot decisions, but establishing whether a memory system realizes this benefit requires fact-specific evidence beyond aggregate success rates.
作者：罗科元, Yi Xie, Wanyun Ling, Yuanzhe Li, Ziyue Li, Zhuoqi Zeng
来源
本页事实对应的公开页面，以原文为准。