外观
论文HF Daily Papers · arXiv 2610.360682 天前
Rollback：世界模型反事实回溯提升机器人策略成功率超10%
本文由 AI 根据公开内容整理，可能有误，请以原文为准：Rollback: Experience Backtracking in World Models for Policy Improvement（HF Daily Papers · arXiv 2610.36068）
核心内容
针对现有世界模型仅作为前向模拟器、无法利用环境回溯特性的局限，研究提出Rollback策略改进范式。该方法将世界模型转化为反事实推理引擎，通过在关键决策点回滚并分支探索替代动作，对比结果以归因成功要素，从而在不依赖独立价值模型的情况下生成动作级监督信号。实验显示，在积木提取、抛球及桌面冰壶三项真实机器人任务中，Rollback相比最强基线成功率分别绝对提升10.67%、11.11%和11.85%，且在8小时内训练即超越人类平均成功率。
系统架构上，Rollback构建了分布式世界模型搜索与异步真机执行的统一训练闭环，实现“测试时物理接地”（test-time physics grounding）。不同于传统手工物理仿真器，该机制利用真机交互数据通过轻量级LoRA流水线实时修正世界模型动力学误差，形成自我强化循环。这种设计不仅解决了模型与现实的对齐问题，还确保了反事实想象集中在对策略优化最关键的决策节点，显著提升了样本效率与训练稳定性，为具身智能从被动模拟转向主动反事实学习提供了新路径。
关键看点
- 反事实引擎：在关键决策点回滚并分支探索替代动作，无需独立价值模型即可生成动作级细粒度监督信号
- 性能提升：在积木提取、抛球及桌面冰壶任务中，成功率较最强基线分别绝对提升10.67%、11.11%和11.85%
- 动态修正：采用轻量级LoRA流水线利用真机数据实时校正世界模型动力学，实现测试时物理接地与自我强化
- 训练效率：尽管初始演示远低于人类水平，仅需8小时训练即在抛球和桌面冰壶任务中超越人类平均成功率
工程与产业观察
Rollback验证了世界模型从被动预测器向主动反事实推理引擎转型的工程价值。其核心启示在于利用虚拟环境的可回溯性弥补真机试错成本，并通过LoRA微调实现Sim-to-Real的在线对齐。这为具身算法开发者提供了摆脱纯前向模拟、构建高样本效率自进化系统的可行技术路线。
英文摘要 (Abstract)
Action-conditioned world models offer a scalable alternative to costly real-world interaction for improving robot policies. However, existing world-model-based methods largely treat world models as forward-only surrogates of the physical environment, leaving untapped a capability no physical environment affords: rewinding to past decision points to explore counterfactual futures. We introduce Rollback, a policy improvement paradigm that turns world models from passive simulators into counterfactual engines: it rolls back to critical decision points, branches into alternative actions, and contrasts their outcomes to attribute success to individual decisions. This concentrates imagination where it matters most and distills counterfactual outcomes into fine-grained, action-level supervision without a separately learned value model. We build a scalable training system that unifies distributed world-model search with asynchronous real-robot execution, forming a self-reinforcing cycle: real-world experience continually sharpens the world model, and a sharper world model in turn drives further policy improvement. We term the world-model side of this cycle test-time physics grounding, a capability that hand-built physics simulators lack: while the world model serves as the training environment, real transitions from the policy it trains correct its dynamics through a lightweight LoRA pipeline. Experiments on three challenging real-world tasks show that Rollback improves success rates over the strongest baseline by an absolute 10.67%, 11.11%, and 11.85% on block extraction, ball tossing, and tabletop curling, respectively, while achieving more stable improvement throughout training. Although initialized from teleoperated demonstrations far below human level, Rollback surpasses the average success rate of human players on both ball tossing and tabletop curling within 8 hours of training.
Hugging Face Daily Papers 热度：50 upvotes 作者：Jiayi Luo, Yang Zhang, Liliang Chen, Nan Wang, Chen Gao, boxiang qiu, Zhuoxuan Li, Baorui Peng, Chuankang Li, Yuxiang Yan, Si Liu, Qingyun Sun, Shuicheng Yan, Maoqing Yao, Jianxin Li, Guanghui Ren
来源
本页事实对应的公开页面，以原文为准。