# ROOT：发现用于用户指定具身行为的奖励

> 原文链接：https://robotics.hellobike.com/papers/2026-10-03-2610-04250
> 频道：papers | 更新日期：2026-10-03 | 标签：arxiv, cs.RO
> 核心摘要：ROOT通过可观察树和视频语言模型引导奖励搜索，使策略对齐用户指定具身行为，并在仿真与真实Unitree Go2等任务上验证。

# ROOT：发现用于用户指定具身行为的奖励

::: v-pre
> 原文链接：[ROOT: Discovering Rewards for User-Specified Embodied Behaviors](https://arxiv.org/abs/2610.04250)（arXiv 2610.04250 · 发布日期：2026-10-03）

**ROOT: Discovering Rewards for User-Specified Embodied Behaviors**

ROOT通过可观察树和视频语言模型引导奖励搜索，使策略对齐用户指定具身行为，并在仿真与真实Unitree Go2等任务上验证。

## 核心要点

- 具身控制中的奖励指定困难，LLM生成奖励常难刻画自然步态、姿态和运动风格等细微行为。
- ROOT将奖励设计表述为观察引导的持久实验树搜索，存储奖励程序、策略、回放观察和行为洞察。
- 视频语言模型从回放中诊断行为失败并指导后续奖励改进，而非仅依赖标量训练统计。
- ROOT较现有LLM奖励方法更符合用户意图，达86.8%运动完整性准确率，人类偏好率51-63%。

**验证范围与局限**：验证范围为七个任务，覆盖仿真Hopper、HalfCheetah、Ant、Unitree Go2以及真实Unitree Go2。

## 英文摘要 (Abstract)

Reinforcement learning for embodied control remains constrained by the difficulty of reward specification. Although recent large language model (LLM)-based methods can synthesize reward functions from natural-language descriptions, they often fail to capture subtle behavioral properties that humans care about, such as natural gait, posture, and movement style. This limitation arises because many desired behaviors are easier to recognize visually than to encode in a reward function. We introduce Reward Optimization via Observable Trees (ROOT), a framework for discovering reward functions that align learned policies with user-specified embodied behaviors. Rather than relying solely on scalar training statistics, ROOT casts reward design as an observation-guided search over a persistent experiment tree that stores reward programs, trained policies, and rollout observations, together with behavioral insights distilled by a video-language model, to diagnose behavioral failures and guide subsequent reward refinements. We evaluate ROOT on seven tasks across four embodiments: simulated Hopper, HalfCheetah, Ant, Unitree Go2, and as well as the real-world Unitree Go2. ROOT produces behaviors that better align with user intent than those generated by existing LLM-based reward-generation methods, achieving up to 86.8% locomotion-completeness accuracy and improving Vid-LLM behavioral alignment from 3.56/5 to 4.14/5, a 16.5% improvement over baselines. Human evaluations further support these results, with ROOT preferred in 51-63% of pairwise comparisons.

作者：Eren Sadikoglu, Aditya Taparia, Xinyuan Liu, Ransalu Senanayake
:::

## 原始参考来源 (Sources)

- [arXiv 2610.04250：ROOT: Discovering Rewards for User-Specified Embodied Behaviors](https://arxiv.org/abs/2610.04250)