# IVG框架实现具身视觉交互闭环评测，SOTA模型任务完成率仅53.4%

> 原文链接：https://robotics.hellobike.com/papers/2026-11-05-2611-08286-88014f14
> 频道：papers | 更新日期：2026-11-05 | 标签：auto, arxiv, hf-daily-papers
> 核心摘要：针对视觉交互模型缺乏闭环评测痛点，提出Interactive Visual Gym框架及含315场景的IVG-Bench。测试显示Gemini 4 Argon任务完成率仅53.4%，且存在延迟与接地缺陷；GRPO实验证实该环境可为Qwen3.5-4B提供有效学习信号。

# IVG框架实现具身视觉交互闭环评测，SOTA模型任务完成率仅53.4%

::: v-pre
> 本文由 AI 根据公开内容整理，可能有误，请以原文为准：[Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds](https://arxiv.org/abs/2611.08286)（HF Daily Papers · arXiv 2611.08286）

## 核心内容

针对现有基准依赖离线回放、无法捕捉助手响应对用户行为动态影响的局限，研究团队推出Interactive Visual Gym框架，支持视觉交互模型的闭环评测与训练。该框架构建了覆盖烹饪、家庭操作及移动界面三大领域的仿真器，并集成角色条件化用户模拟工具以引入受控执行误差。基于此实例化的IVG-Bench包含315个交互场景，经人类研究验证，其交互质量指标与用户偏好一致性达78.5%。

在IVG-Bench上评估SOTA VLM发现，表现最优的Gemini 4 Argon限时任务完成率仅为53.4%，且在事实接地、可操作指导及响应延迟方面存在显著短板。同时，环境被验证能提供有效学习信号：在烹饪任务的GRPO概念验证实验中，闭环训练成功提升了Qwen3.5-4B模型的任务成功率与交互质量，表明该仿真体系兼具高保真评测与具身算法迭代双重价值。

## 关键看点

- **闭环仿真架构**：构建烹饪、家庭操作及移动界面三类仿真器，集成角色条件化用户模拟工具，支持引入受控执行误差与非计划行为，解决离线评测无法反映动态交互的难题。
- **SOTA模型瓶颈**：IVG-Bench测试显示最强模型Gemini 4 Argon限时任务完成率仅53.4%，暴露出事实接地能力弱、行动指导模糊及响应延迟导致纠错失败等关键工程缺陷。
- **训练信号验证**：在烹饪环境GRPO实验中，闭环训练有效提升Qwen3.5-4B的任务成功率与交互质量，证实该仿真平台可作为具身视觉交互模型强化学习的有效数据源。
- **评测协议对齐**：通过人类对照研究验证自动化评测有效性，系统交互质量指标与真实参与者偏好的一致性达到78.5%，确保了基准测试结果对实际部署的参考价值。

## 工程与产业观察

该工作填补了具身视觉交互从离线评测迈向闭环验证的关键缺口。对于算法工程师，IVG-Bench揭示了当前VLM在动态物理交互中的真实性能天花板与延迟敏感性；其验证的GRPO训练范式则为利用仿真数据优化端侧交互模型提供了可复现路径，是构建高可靠人机协作系统的必要基础设施。

## 英文摘要 (Abstract)

Visual interaction models that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users. We contribute three simulators that recreate everyday tasks and human-aligned action spaces across cooking, household manipulation, and mobile-interface domains, together with a persona-conditioned user simulation harness that collaborates with the assistant while incorporating controlled execution errors and other out-of-plan behaviors. We instantiate the framework as IVG-Bench, a benchmark with 315 interaction scenarios. Evaluating state-of-the-art VLMs as visual interaction models reveals substantial limitations: the best-performing model, Gemini 4 Argon, completes only 53.4% of tasks within the time limit, exhibits significant weaknesses in factual grounding and actionable guidance, and is often too slow to prevent errors when response latency is considered. We verify our evaluation protocol with a human study which finds 78.5% agreement between our interaction-quality metric and participant preferences. We further show that the environments provide useful learning signals: in a proof-of-concept GRPO experiment, closed-loop training improves both task success and interaction quality for Qwen3.5-4B model in our cooking environment.

Hugging Face Daily Papers 热度：1 upvotes
作者：Chenhao Zheng, Zixian Ma, Kai Li, Kanchana Ranasinghe, Jieyu Zhang, Artemis Panagopoulou, Weikai Huang, Jianxu Shangguan, Yuanyuan Tan, Silvio Savarese, Juan Carlos Niebles, Ranjay Krishna, Ran Xu, Ho…
:::

## 原始参考来源 (Sources)

- [HF Daily Papers · arXiv 2611.08286：Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds](https://arxiv.org/abs/2611.08286)