# 无需高质量状态估计的真实机器人交互强化学习

> 原文链接：https://robotics.hellobike.com/papers/2026-10-02-2610-14042
> 频道：papers | 更新日期：2026-10-02 | 标签：arxiv, cs.RO
> 核心摘要：针对真实机器人RL中机载状态估计不准的问题，提出RSSM-RLPD，并在NAO带球射门真机与仿真中验证。

# 无需高质量状态估计的真实机器人交互强化学习

::: v-pre
> 原文链接：[Reinforcement Learning from Real-Robot Interaction without High-Quality State Estimates](https://arxiv.org/abs/2610.14042)（arXiv 2610.14042 · 发布日期：2026-10-02）

**Reinforcement Learning from Real-Robot Interaction without High-Quality State Estimates**

针对真实机器人RL中机载状态估计不准的问题，提出RSSM-RLPD，并在NAO带球射门真机与仿真中验证。

## 核心要点

- 真实物理经验学习常受部分可观测影响，状态估计误差会污染策略输入和奖励。
- 方法用UKF估计关键状态，并以RSSM学习状态表征，仅在观测到球时校正潜变量。
- 在NAO带球进门任务中，30分钟真实交互后可从多个起点学会得分，成功率100%。
- 仿真研究中，RSSM-RLPD相较行为克隆、前馈和GRU基线取得更高成功率。

**验证范围与局限**：真机验证为一台NAO机器人带球进门任务，并辅以仿真研究。

## 英文摘要 (Abstract)

A longstanding goal for robot reinforcement learning (RL) is that robots learn from their own physical experience, as humans and animals do. A key obstacle to learning directly from physical experience, especially outside instrumented lab environments, is that the state of the world is usually unobservable to the learner and must be estimated from on-board sensing. Errors in that estimate corrupt both the policy input and potentially the reward signal. We study this challenge using the contact-rich task of a bipedal robot learning to score soccer goals. We estimate all key state variables on-board with an unscented Kalman filter (UKF) and no external instrumentation (e.g., motion capture or overhead cameras). Because the UKF may incorrectly estimate state when the ball is unobserved, we learn a new state representation with a recurrent state-space model (RSSM) that, critically, only corrects the RSSM's latent state when the ball is observed. Building on RLPD, we introduce RSSM-RLPD and apply it to the task of a NAO robot learning to dribble a soccer ball into a goal. We show that the robot learns to score from multiple starting positions after 30 minutes of real-world interaction, reaching 100% success against 0% for a feed-forward policy under the same training budget. We complement our physical robot experiments with a simulation study and show that RSSM-RLPD learns policies that achieve a higher success rate than behavior cloning, a feed-forward, and GRU baseline. Taken together, our results show that the addition of a learned state representation to existing on-robot learning methods enables RL from real-robot data even with a high degree of partial observability.

作者：Ryan Gao, Ashvin Anilkumar, Nicholas Corrado, William Cong, Josiah P. Hanna
:::

## 原始参考来源 (Sources)

- [arXiv 2610.14042：Reinforcement Learning from Real-Robot Interaction without High-Quality State Estimates](https://arxiv.org/abs/2610.14042)