# 稳定控制中零阶奖励塑形对策略梯度的充分性

> 原文链接：https://robotics.hellobike.com/papers/2026-09-28-2609-34695
> 频道：papers | 更新日期：2026-09-28 | 标签：arxiv, cs.RO
> 核心摘要：研究稳定控制中奖励需包含的信息，证明策略梯度用零阶塑形即可求解，并在稳定任务上理论与经验验证。

# 稳定控制中零阶奖励塑形对策略梯度的充分性

::: v-pre
> 原文链接：[Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control](https://arxiv.org/abs/2609.34695)（arXiv 2609.34695 · 发布日期：2026-09-28）

**Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control**

研究稳定控制中奖励需包含的信息，证明策略梯度用零阶塑形即可求解，并在稳定任务上理论与经验验证。

## 核心要点

- 现代机器人深度RL奖励塑形仍依赖启发式，难区分目标项与数值正则。
- 研究稳定控制中奖励必须包含的量，聚焦零阶配置与一阶速度信息。
- 理论和经验表明，无一阶奖励项策略梯度仍可解稳定任务；加入后尺度大时更敏感。
- 目标相关坐标需零阶完备；在低耗散假设下，策略观测仍需一阶状态。

**验证范围与局限**：验证限于摘要所述稳定控制任务的理论与经验研究，未说明仿真或真机平台。

## 英文摘要 (Abstract)

Reward shaping is fundamental to modern robotic control with deep reinforcement learning (RL), yet practitioners still rely heavily on heuristic principles borrowed from classical optimal control and trajectory optimization. Existing methods rarely distinguish reward terms that are intrinsic to the control objective from numerical regularizers, leading to brittle hyperparameter tuning. To determine which quantities a reward must contain, we study the stabilization control problem with a focus on zeroth-order (configuration) and first-order (velocity) information. We theoretically and empirically demonstrate that policy gradient methods can successfully solve stabilization tasks without first-order reward terms, adding such terms can instead introduce severe sensitivity as their scale grows. Conversely, our findings confirm that reward functions must be zeroth-order complete over goal-relevant coordinates, while the first-order state remains necessary in the policy observation under our low-dissipation assumptions. Overall, these results provide actionable and principled guidance for reward design in robotic RL.

作者：Yisheng Zhang, Tao Wang, Sicun Gao
:::

## 原始参考来源 (Sources)

- [arXiv 2609.34695：Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control](https://arxiv.org/abs/2609.34695)