论文arXiv 2611.1043012 天前
信念空间规划器蒸馏策略在Scavenger任务成功率超90%
本文由 AI 根据公开内容整理,可能有误,请以原文为准:Deploying Physically and Epistemically Compliant Agents by Distilling Belief-Space Task and Motion Planners(arXiv 2611.10430)
核心内容
针对深度强化学习难以完美遵循规则、而信念空间规划器在线部署工程复杂度高的问题,该研究提出离线生成轨迹数据集并蒸馏为基于原始观测的策略。通过引入Dual Averaging损失函数与ρ-Optimal Reinforcement反馈机制,解决了历史依赖策略的特有难题,并在静态搜索问题上证明了无噪观测模型下的完备性条件。
在新构建的Scavenger信息获取测试域中,蒸馏策略在视觉与自然语言两种信息模态下成功率均超90%,对比纯RL方法不足5%的性能实现数量级提升。泛化测试显示,该策略对未见过的模糊自然语言指令保持超60%执行成功率,在大于训练尺寸的楼层平面图中迁移成功率超75%,验证了从规划器到端到端策略的有效知识传递。
关键看点
- 蒸馏算法创新:提出Dual Averaging损失结合时序预测与任务变更平均损失,辅以ρ-Optimal Reinforcement机制强化近优自采样轨迹
- Scavenger基准性能:在需主动获取信息的视觉与语言双模态任务中,蒸馏策略成功率突破90%,纯RL方法上限仅5%
- 零样本泛化能力:面对训练中未见的模糊自然语言指令,策略执行成功率仍超60%,验证语义理解与规划逻辑的解耦迁移
- 空间尺度迁移:在面积大于训练集的楼层平面图中,策略导航与搜索成功率保持75%以上,具备环境拓扑扩展适应性
工程与产业观察
该工作验证了“规划器生成数据+策略蒸馏”范式在具身智能长程推理任务中的有效性,解决了RL在安全合规与逻辑遵循上的短板。对于机器人开发者,这意味着可将高可靠但昂贵的在线TAMP系统转化为轻量级端到端模型,在保证行为合规的前提下大幅降低端侧推理成本,是连接符号规划与神经网络控制的关键工程路径。
英文摘要 (Abstract)
In this work, we train agents to act under uncertainty according to specified, belief-dependent rules. Deep reinforcement learning (RL) can in theory train agents to follow rules if encoded into well-designed rewards, but in practice, RL fails to perfectly optimize these rewards, leading to ineffective or noncompliant agent behavior. Belief-space planners by construction produce behavior in line with developer specification, but require hand-programming online planners and belief-state estimators, adding significant engineering complexity at scale. We combine the two: we plan offline to generate a dataset of sample trajectories and then distill the plans into a policy that acts on raw observations. We propose two changes to planner distillation that address problems specific to history-based policies: Dual Averaging, a loss that combines per-timestep predictive loss and average loss on task changes, and $\rho$-Optimal Reinforcement, a feedback mechanism that reinforces the policy's own successful rollouts when these are near-optimal. We provide a sufficient condition under which static search problems with noiseless observation models is complete, and show that maximum-likelihood-observation planning implies this condition in static search with continuous beliefs. We create a challenging new domain, Scavenger, where an agent needs to gain information in order to succeed. We evaluate our distilled policy in two versions of Scavenger, one where information is provided visually and one where it is provided as a command in natural language. In either case, the learned policy attains a success rate above 90% in Scavenger, compared with at most 5% for RL. We also find that distilled planners transfer with greater than 60% success to following unseen and intentionally vague commands and greater than 75% success to following floorplans larger than those seen in training.
作者:Phillip J. K. Christoffersen, Rachel Ma, Patrick Barrag'an, Dylan Hadfield-Menell
外部来源
本页事实对应的公开页面,以原文为准。(点击将离开本站)