外观
论文HF Daily Papers · arXiv 2610.314946 天前
AgentVQA基准发布：22个VLM在具身场景最高准确率仅58.5%
本文由 AI 根据公开内容整理，可能有误，请以原文为准：AgentVQA: A Unified Benchmark for Agentic Visual Understanding（HF Daily Papers · arXiv 2610.31494）
核心内容
为解决在线仿真评测成本高且难扩展的问题，研究团队发布AgentVQA离线基准，覆盖Web、机器人、第一人称视频、游戏及空间规划五大具身领域。该基准从23个源数据集中人工筛选并适配了1000个高难度真实交互样本，旨在提供可靠且可扩展的视觉语言模型（VLM）评估标准。实测22款主流模型，最高准确率仅为58.5%，揭示了当前VLM在复杂具身任务中的性能天花板。
跨域分析表明，模型失败主因并非任务误解或下游逻辑推理缺陷，而是无法准确接地并维持对当前及演化视觉状态的表征。具体技术短板包括UI元素定位失效、运动语义解析错误、时序状态追踪丢失以及3D空间关系推断失败。验证结果显示，AgentVQA的离线排名与在线交互评测高度一致，证明其能有效替代昂贵的在线测试，精准诊断具身视觉感知系统的核心工程瓶颈。
关键看点
- 基准构建：从23个数据集精选1000条真实样本，覆盖机器人等五大具身领域，支持低成本离线评测
- 性能上限：22个VLM模型中最高准确率仅58.5%，量化了当前多模态模型在真实交互场景的能力缺口
- 核心瓶颈：错误源于视觉状态接地与时序追踪失效，而非逻辑推理，指明感知算法优化方向
- 评测效度：离线基准排名与在线仿真/真机评测结果高度对齐，验证了其作为具身模型筛选工具的可靠性
工程与产业观察
AgentVQA以量化数据证伪了“大模型即通用具身大脑”的假设，明确当前VLM在动态视觉接地与3D时空推理上存在系统性缺陷。这提示产业界在部署具身智能时，不能仅依赖端到端大模型的涌现能力，需针对性强化视觉状态估计与空间表征模块，或将该基准作为选型与训练数据清洗的关键标尺。
英文摘要 (Abstract)
Vision-language models (VLMs) can perform a broad range of tasks across diverse settings, yet their performance in agentic contexts remains poorly understood. Although online evaluations in simulators and real-world environments are essential for measuring interactive agent performance, they are costly and challenging to scale. Complementary offline VQA benchmarks can enable faster and more reliable comparisons across agentic domains, yet existing benchmarks remain limited and fragmented. To address this gap, we introduce AgentVQA, an offline benchmark for evaluating VLMs on realistic agentic scenarios across five agent domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning. We build AgentVQA by manually selecting 1000 challenging, real-world agent examples from 23 source datasets and adapting them for reliable offline evaluation. The resulting benchmark preserves realistic scenarios from interactive environments across diverse domains, and its model rankings track those from corresponding online evaluations. Across the 22 evaluated models, the best achieves only 58.5% accuracy. Our cross-domain analysis shows that errors are driven primarily by failures to ground and maintain accurate representations of the current and evolving visual state, including localizing relevant UI elements, interpreting motion, tracking state changes over time, and inferring 3D spatial relationships, rather than by task misinterpretation or downstream logical reasoning.
Hugging Face Daily Papers 热度：17 upvotes 作者：Advait Gupta, Sachin Konan, Tom Yin, Gabriel Sarch, Zhuang Liu
来源
本页事实对应的公开页面，以原文为准。