外观
论文arXiv 2609.327521 个月前
Action Shaping：策略吸收其可表达之物
原文链接：Action Shaping: Policies Absorb What They Can Express（arXiv 2609.32752 · 发布日期：2026-09-26）
Action Shaping: Policies Absorb What They Can Express
针对训练加动作偏移、部署移除无保证的问题，提出action shaping条件，用门控线性头促使策略吸收可表达偏移，并在20个任务上验证。
核心要点
- 动机是奖励塑形有势函数定理，而动作通道偏移训练后删除缺少保证。
- 方法是在经学得动作价值函数训练的actor上，加入零初始化线性头与可学习门控。
- 实验显示门控会自行升降；在20个任务中移除线性头几乎不损失回报。
- 吸收取决于精确复现而非容量；偏移幅度可在移除前指示代价。
验证范围与局限：验证范围限于摘要所述20个任务及确定性、随机actor设置；未说明仿真或真机。
英文摘要 (Abstract)
Reward shaping has a theorem: a potential-based term can be removed without changing the optimal policy. The same practice on the action channel, an offset added in training and dropped at deployment, has no theorem. Nothing cancels an action offset, so the correction is kept at deployment or removed without a guarantee. We call it action shaping and state its principle. A trainable policy absorbs an offset its own output layer can reproduce exactly, which is what we mean by express; what is absorbed can be removed with the return intact. Its minimal instance is a zero-initialized linear head behind a learnable gate, added to an actor that trains through a learned action-value function, with no penalty or schedule. The gate rises and then falls on its own, for deterministic and stochastic actors alike, and on 20 tasks removing the head costs almost nothing. The condition is exact reproduction, not capacity: a nonlinear head with more parameters is not absorbed, and in a paired control, one linear path added to a nonlinear base head restores absorption. Exact reproduction gives the loss a flat direction that gradient noise drifts along, and the offset's amplitude indicates, before removal, what dropping the head will cost. Action shaping thus gains the counterpart of the shaping theorem, a condition for absorption, together with the mechanism behind it and a diagnostic that reads it. Policies absorb what they can express, and only that.
作者：Yanjun Chen, Jinghan Wang, Xiaoyu Shen, Wenjie Li, Wei Zhang
来源
本页事实对应的公开页面，以原文为准。