论文HF Daily Papers · arXiv 2611.024972 天前

RoboColosseum分布式评测：5699次实测揭示VLA策略跨本体泛化瓶颈

该框架联合124个站点完成5699次盲测，评估5种VLA与3种AI Agent在SO-101、Franka等4类本体上的表现。结果显示Open Track排名无法预测微调后性能，G1人形机器人任务完成率为零但进度差异显著，证实分布式物理评测对量化具身智能泛化能力的必要性。

AI 整理

具身模型
评测与安全

宇树 G1 Franka FR3 pi0_base xvla-agibot-world

本文由 AI 根据公开内容整理，可能有误，请以原文为准：RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild（HF Daily Papers · arXiv 2611.02497）

核心内容

RoboColosseum提出一种结合统一策略服务与标准化协议的分布式评测框架，旨在解决具身智能物理试验难复现问题。研究涵盖5种视觉-语言-动作（VLA）策略及3种前沿AI Agent，在SO-101、Franka、Bimanual-YAM及G1 Humanoid四种本体上开展测试。通过124个站点的5699次成对盲测及每任务100条示教微调，发现前沿Agent在开放赛道可比肩专用VLA，但所有系统在开放世界任务中可靠性均不足。

技术层面，微调后GR00T N1.7在SO-101领先，pi0.5在Franka与双臂YAM占优，但各本体任务套件排名一致性极低（Kendall W=0.24）。Open Track排名与微调后性能存在大幅反转，表明零样本能力不代表适配潜力。针对G1 Humanoid五项任务虽无完全成功，但部分进度指标区分度达13.3%至23.2%。此外，主动匹配机制以更少试验量实现了与均匀采样相当的排名稳定性。

关键看点

跨本体排名低一致性：四种本体特定任务套件的策略排名Kendall W系数仅为0.24，证实单一基准无法衡量通用物理智能。
零样本与微调性能解耦：Open Track排名无法预测微调后表现，三个共享本体均出现显著排名反转，凸显适配数据的重要性。
部分进度量化失效任务：G1 Humanoid五项任务完成率为0，但策略间平均进度介于13.3%至23.2%，揭示了二值成功率掩盖的能力差异。
主动匹配提升评测效率：相比均匀采样，主动匹配算法在大幅减少物理试验次数的同时，保持了同等的排名收敛性与一致性。

工程与产业观察

该研究揭示了当前VLA模型跨本体泛化的真实瓶颈：零样本推理能力与少样本适配能力并不正相关，且不同硬件形态下的最优策略高度异构。对于具身算法研发，仅靠单一仿真或单机Benchmark已失效，必须建立包含部分进度度量与分布式实机验证的评测体系，才能准确指导数据飞轮构建与模型选型。

英文摘要 (Abstract)

General-purpose robot policies must work across robots, tasks, and environments, yet the variation that makes physical trials informative also makes them difficult to reproduce. We introduce RoboColosseum, a distributed evaluation framework that combines unified policy serving and standardized protocols across a network of locally operated robots in diverse environments. Our initial study evaluates five vision-language-action (VLA) policies and three frontier AI agents across four embodiments (SO-101, Franka, Bimanual-YAM, and G1 Humanoid). The Open Track compares models without task-specific adaptation in blinded, pairwise trials under matched local conditions. The Fine-tune Track measures how well VLA policies adapt to 20 tasks across all four embodiments when each policy receives the same 100 demonstrations per task. Across 5,699 pairwise Open Track comparisons at 124 sites, frontier agents rival specialized VLA policies, while all systems struggle to complete open-world tasks reliably. After fine-tuning, GR00T N1.7 leads on SO-101 and pi0.5 leads on Franka and Bimanual-YAM. Policy rankings show weak agreement across embodiment-specific task suites (Kendall W = 0.24). Open Track rankings poorly predict relative performance after fine-tuning, with substantial rank reversals on all three embodiments shared between tracks. Measuring partial progress reveals differences that task success alone misses. On the G1 Humanoid, no policy completes any of the five tasks, yet average progress ranges from 13.3% to 23.2% across policies. Open Track rankings stabilize as trials accumulate, and active matchmaking achieves comparable ranking agreement with substantially fewer physical trials than uniform sampling. These results show how distributed physical evaluation can reveal where robot policies generalize, how well they adapt to new tasks, and how far they progress on tasks they fail to complete.

Hugging Face Daily Papers 热度：34 upvotes 作者：Tan Sang Nguyen, Yi Ru Wang, Shailesh Shailesh, Gang Yang, Jiamo Hu, Jianuo Jiang, Haoquan Fang, Zhongzheng Ren, Zhongyi Luo, Ian Chong, Aaron Tung, Santiago Pravisani, Minghui Lin, Yao Mo, Ishneet Si…