# 重新审视深度强化学习中用于平滑控制的时间正则化

> 原文链接：https://robotics.hellobike.com/papers/2026-10-06-2610-07910
> 频道：papers | 更新日期：2026-10-06 | 标签：arxiv, cs.RO
> 核心摘要：针对深度强化学习动作振荡，提出结合时间惩罚与线性渐增的CATS，并在仿真和真实世界验证平滑控制。

# 重新审视深度强化学习中用于平滑控制的时间正则化

::: v-pre
> 原文链接：[Revisiting Temporal Regularization for Smooth Control in Deep Reinforcement Learning](https://arxiv.org/abs/2610.07910)（arXiv 2610.07910 · 发布日期：2026-10-06）

**Revisiting Temporal Regularization for Smooth Control in Deep Reinforcement Learning**

针对深度强化学习动作振荡，提出结合时间惩罚与线性渐增的CATS，并在仿真和真实世界验证平滑控制。

## 核心要点

- 深度强化学习策略可能产生不平滑动作振荡，阻碍在物理机器人上部署。
- 论文证明时间惩罚可约束共享下一状态的当前状态间期望动作差，呈现空间效应。
- CATS将时间惩罚与线性渐增结合，使策略先学习高回报行为再逐步平滑动作。
- 仿真和真实世界实验表明，CATS显著减少动作振荡且不降低任务性能，计算开销小。

**验证范围与局限**：验证范围为仿真和真实世界实验，摘要未说明具体机器人或任务。

## 英文摘要 (Abstract)

Deep Reinforcement Learning policies can produce nonsmooth action oscillations that hinder deployment on physical robots. Existing architectural and penalty-based approaches seek spatial smoothness by directly reducing sensitivity to changes in state inputs, but their broad constraints can degrade task performance as stronger smoothing is pursued. Temporal regularization instead constrains action differences along observed transitions, but has been considered unable to provide the spatial smoothness needed under observation noise. We revisit this assumption by proving that the temporal penalty bounds the expected action differences between current states sharing a next state, revealing a spatial effect that empirically extends to spatial smoothness. Building on this finding, we propose Conditioning for Action using only Temporal Smoothness (CATS), which combines a temporal penalty with linear ramp-up. We highlight temporal regularization's ability to provide spatial smoothness while better preserving task performance than explicit spatial regularization. Through linear ramp-up, CATS allows the policy to learn rewarding behavior before progressively smoothing its actions, improving return preservation and both temporal and spatial smoothness. Experiments in both simulation and the real world show that CATS substantially reduces action oscillation without degrading task performance, with little computational overhead.

作者：SungJae Ahn, Jeong Woon Lee, Kyoleen Kwak, Hyoseok Hwang
:::

## 原始参考来源 (Sources)

- [arXiv 2610.07910：Revisiting Temporal Regularization for Smooth Control in Deep Reinforcement Learning](https://arxiv.org/abs/2610.07910)