具身开发者社区

外观

论文arXiv 2610.1295719 天前

JoyInside-VLM-Agent：面向具身智能体的产品条件化多模态策略学习 ​

提出JoyInside-VLM-Agent，通过产品条件化轨迹构建与分阶段策略学习，在机器人平台、智能硬件和公开基准上验证多模态动作生成。

AI 整理
  • 硬件零部件
  • 本体 SDK 与控制
  • 具身模型

原文链接：JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agents（arXiv 2610.12957 · 发布日期：2026-10-09）

JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agents

提出JoyInside-VLM-Agent，通过产品条件化轨迹构建与分阶段策略学习，在机器人平台、智能硬件和公开基准上验证多模态动作生成。

核心要点 ​

  • 具身智能体需将多模态观察和用户请求转为动作，并适应产品约束、目标变化和反馈。
  • 方法构建ReAct式多模态智能体，用闭环沙盒生成轨迹，并经微调、蒸馏和DPER强化学习训练。
  • 在BFCL v4总体得分77.44，高于所评基线；其他公开基准结果不一。
  • 在JoyInside-Devices和JoyInside-Robotics上，含领域技能细化的系统各指标领先基线。

验证范围与局限：验证范围包括公开基准、JoyInside-Devices/JoyInside-Robotics，以及两个真实世界四足案例研究。

英文摘要 (Abstract) ​

Embodied agents must translate multimodal observations and user requests into executable actions while adapting to product constraints, changing goals, and execution feedback. We present JoyInside-VLM-Agent, a ReAct-style multimodal agent developed through product-conditioned trajectory construction and staged policy learning for robotic platforms and smart hardware. A closed-loop sandbox generates multimodal interaction trajectories covering goal revisions and failure recovery, which are checked against execution outcomes and semantic criteria. Supervised fine-tuning and multi-teacher on-policy distillation precede reinforcement learning with DPER (Dual-Source Privileged On-Policy Self-Distillation with Token-Level Evidence Routing). DPER routes token-level evidence between a frozen privileged teacher and a privileged student sharing the rollout policy's parameters, modulating reward-derived token advantages while preserving their signs. Training rewards combine generative process evaluation, multi-judge rankings, and task-queue consistency. JoyInside-VLM-Agent achieves a BFCL v4 overall score of 77.44, exceeding the evaluated baselines; results on the other public benchmarks are mixed. On JoyInside-Devices and JoyInside-Robotics, the evaluated system, which includes domain-specific skill refinement, leads the baselines on all reported metrics. Two real-world quadruped case studies illustrate visually grounded search, replanning after execution failure, and explicit failure reporting.

作者：Gangyi Li, LuYao Wang, Jingfu Sun, Jingyi Yuan, Hang Liu, Hao Zong, Yujia Hu, Tingting Gao, Jianyong Wang, Xu Yang

来源

本页事实对应的公开页面，以原文为准。

  1. arXiv 2610.12957：JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agents访问 2026-10-21