外观
论文arXiv 2610.1302419 天前
World Action Models需要什么视觉表征？
原文链接：What Visual Representations Do World Action Models Need?（arXiv 2610.13024 · 发布日期：2026-10-09）
What Visual Representations Do World Action Models Need?
针对WAM中VAE偏重重建而非操作的问题，研究比较视觉表征与集成机制，并在RoboCasa GR1 Tabletop和RoboTwin 2.0验证R²-WAM。
核心要点
- WAM继承视频生成先验，但VAE表征主要优化视觉重建而非操作。
- 在受控设置下系统比较视觉表征、五种集成机制与特征组合。
- 强通用表征更鲁棒；通道级联合预测获得最高平均成功率。
- R²-WAM在两个基准上达84.7%和92.8%，较VAE基线提升5.9和2.2个百分点。
验证范围与局限：验证范围限于摘要提到的RoboCasa GR1 Tabletop和RoboTwin 2.0基准。
英文摘要 (Abstract)
World Action Models (WAMs) inherit powerful priors from pretrained video generation models, but their variational autoencoder (VAE) representations are optimized for visual reconstruction rather than manipulation. External visual representations can improve manipulation, but existing methods couple representation choices with different integration mechanisms, obscuring their individual effects. We systematically compare visual representations, five integration mechanisms, and feature combinations under controlled settings. Our study yields three main findings. First, strong general-purpose visual representations provide broader robustness gains and can match or outperform representations pretrained for specific visual capabilities. Second, channel-wise joint prediction achieves the highest average success rate; further analyses suggest that future-feature prediction and the action expert's effective use of added visual information contribute to its gains. Third, additional sources should be selected for the task-relevant information they contribute beyond existing representations, as the same source can help one combination but harm another. However, even the best evaluated composition provides a smaller additional gain than the initial improvement from adding a strong general-purpose representation to the VAE-only baseline. We instantiate the resulting Representation Recipe as R$^2$-WAM. It achieves state-of-the-art performance on RoboCasa GR1 Tabletop with an 84.7% success rate and reaches 92.8% on RoboTwin 2.0, improving over the VAE-only baselines by 5.9 and 2.2 percentage points, respectively. Taken together, these findings suggest that future work should prioritize stronger general-purpose visual representations and more effective mechanisms for integrating them into WAMs, with feature composition serving as a supplementary route to improvement.
作者：Kai Liu, Jiangran Lyu, Tingyu Deng, Yixin Zheng, Jingtian Wan, Wenbo Cui, Mi Yan, Yizhou Zhou, Haoran Li, Dongbin Zhao
来源
本页事实对应的公开页面，以原文为准。