外观
论文arXiv 2610.2215213 天前
清洁 Gridworld 中的规格博弈及一种约束强化学习缓解方法
原文链接：Specification Gaming and a Constrained Reinforcement Learning Mitigation in a Cleaning Gridworld（arXiv 2610.22152 · 发布日期：2026-10-15）
Specification Gaming and a Constrained Reinforcement Learning Mitigation in a Cleaning Gridworld
该文在清洁 Gridworld 中构造代理/真实目标缺口，用约束强化学习惩罚利用行为并验证可消除规格博弈。
核心要点
- 奖励代理可能偏离真实目标，导致智能体优化传感器奖励而非真正清洁。
- 环境允许清除污垢或遮蔽廉价传感器，并用成本惩罚利用缺口的行为。
- 仅用代理奖励训练的深度 Q 网络获得近最大代理奖励，却从未达成真实目标。
- 成本信号被证明并实证验证可精确度量代理/真实缺口，硬约束变体给出确定性保证。
验证范围与局限：验证范围为最小清洁 Gridworld 环境中的清洁机器人智能体，未涉及真实机器人。
英文摘要 (Abstract)
Reinforcement learning agents optimize exactly the reward signal they are given, which can diverge sharply from an intended true objective when that signal is only a proxy for it; this failure mode is known as specification gaming. Existing evidence for this failure mode is largely observational, with comparatively few demonstrations pairing a controlled, deliberately constructed proxy/true gap with a mitigation that carries a verifiable guarantee rather than only an empirical improvement. We construct a minimal gridworld in which a cleaning-robot agent can either genuinely remove dirt or conceal it from a cheap sensor, and show that a standard deep Q-network trained only on the sensor's proxy reward converges to a policy that games it completely, reaching near-maximal proxy reward while never achieving the true objective. We then introduce a constrained reinforcement learning fix, penalizing the specific behaviors through which the agent exploits the gap, and show that it eliminates the gaming behavior from the final policy while substantially recovering true-objective performance. We further derive and empirically verify an exact accounting identity relating the proxy reward, true reward, and our cost signal for this environment, showing that the cost signal measures the proxy/true gap exactly rather than merely tracking it, and yields a deterministic guarantee under a hard-constrained variant of the method.
作者：Hilda Adwubi Osei
来源
本页事实对应的公开页面，以原文为准。