外观
论文arXiv 2610.2228513 天前
提出更聪明的问题：面向欠规范奖励的定向主动偏好学习
原文链接：Asking Smarter Questions: Targeted Active Preference Learning for Underspecified Rewards（arXiv 2610.22285 · 发布日期：2026-10-15）
Asking Smarter Questions: Targeted Active Preference Learning for Underspecified Rewards
针对人类示教难以覆盖全部任务特征的问题，提出定向偏好查询与解释方法，并在仿真和用户研究中验证。
核心要点
- 人类注意力有限，示教往往对某些任务特征监督不足，导致奖励学习欠规范。
- 方法利用示教数据中暴露的不确定性，构建先验以定向选择需澄清的偏好查询。
- 每个查询配套解释，引导人类关注相关特征，降低再次示教的负担。
- 仿真中低查询场景较标准主动偏好学习最高提升30.8%，用户研究显示先验与解释均有显著作用。
验证范围与局限：验证范围包括仿真实验和用户研究，摘要未说明具体任务或真实机器人实验。
英文摘要 (Abstract)
Reward learning typically assumes the human's input covers everything that matters for the task. In practice, humans can only attend to so much at once, so their teaching naturally captures some features---or aspects of the task---better than others. Prior work has shown how the robot can identify which features were poorly supervised directly from its teaching data. But the current solution is to ask the human to try again, putting the burden on someone who already struggled to convey those features. We propose shifting this burden to the robot. Our key insight is that the same information that tells the robot what it doesn't know can help it ask better questions. Preference queries offer a low-effort channel for resolving this uncertainty, but only if the robot knows where to direct them and the human knows what to attend to when answering. We act on this insight in two ways: we build an informed prior that targets preference queries at the features that need resolving, and we accompany each query with an explanation that focuses the human's attention on exactly those features. In simulation, the informed prior beats the standard active preference learning baseline by up to $30.8%$ in the low-query regime, where reducing human effort matters most. In a user study, the informed prior and explanations each significantly improve reward recovery and amplify each other.
作者：Helena Merker, Nick Walker, Andreea Bobu
来源
本页事实对应的公开页面，以原文为准。