外观
论文arXiv 2610.1540117 天前
在具身RL中利用VLM：通过探索增强关键观察门控从奖励回归器到成功检测器
原文链接：Harnessing VLM in Embodied RL: From Reward Regressor to Success Detector via Exploration-Augmented Key-Observation Gating（arXiv 2610.15401 · 发布日期：2026-10-11）
Harnessing VLM in Embodied RL: From Reward Regressor to Success Detector via Exploration-Augmented Key-Observation Gating
针对从零开始具身RL奖励设计，VLMeg将VLM用作成功检测器，并在MetaWorld和真实实验中验证。
核心要点
- 现有VLM奖励回归需大量标注，且更适合微调预训练策略，从零RL表现较弱。
- VLMeg将VLM对齐为成功检测器，以探索增强关键观察门控产生回合奖励。
- 门控先筛选值得判断的回合，再选择混淆最低关键观察供VLM评估，减少冗余调用。
- 在MetaWorld的12个无奖励任务和3个真实实验中优于先进VLM基线，代码接收后开源。
验证范围与局限：验证范围包括MetaWorld仿真中的12个无奖励任务，以及3个无真实奖励的真实世界实验。
英文摘要 (Abstract)
VLMs provide promising solutions to embodied reward design. Most current methods that train VLMs as reward regressors not only require extensive annotations but are also primarily designed for finetuning pretrained policies, and perform less well in the more challenging from-scratch RL setting. This issue stems from a fundamental mismatch: regressing precise rewards is misaligned with what VLMs naturally excel at. We posit that it is more aligned with VLMs' inherent multi-modal VQA capabilities for the models to function as success detectors, and propose VLM success reward via Exploration-Augmented Key-Observation Gating (VLMeg) to well harness VLM for embodied reward design. As the core of VLMeg, the proposed exploration-augmented key-observation gating first determines whether the current RL episode is judgment-worthy based on its similarity to a generated success trajectory. Then, the key observation with the lowest confusion from the judgment-worthy episode will be selected for VLM evaluation as episodic reward. This gating mechanism not only avoids redundant calls but also enables higher detection accuracy of VLM. Considering the sparsity of judgment-worthy episodes, the advanced exploration incentives based on the generated trajectory are introduced, enabling more efficient exploration toward judgment-worthy trajectories for more VLM reward triggers and thus gaining higher RL efficiency. Extensive experiments in MetaWorld demonstrate VLMeg achieves state-of-the-art from-scratch RL, outperforming advanced VLM-based baselines on 12 challenging reward-free tasks. Furthermore, VLMeg also enables effective RL without any ground-truth reward in 3 challenging real-world experiments. Code will be open after acceptance at https://github.com/Raina915/VLMeg.
作者：Xin Liu, Yutong Jiang, Jian Su, Ziheng Wang, Chuang Wang, Haoran Li, Dongbin Zhao
来源
本页事实对应的公开页面，以原文为准。