外观
论文HF Daily Papers · arXiv 2610.301207 天前
ALPS视觉状态编码器统一预测指令与检视，LIBERO策略成功率达85.5%
本文由 AI 根据公开内容整理，可能有误，请以原文为准：ALPS: Visual State Encoders for World Models（HF Daily Papers · arXiv 2610.30120）
核心内容
针对现有世界模型视觉编码器功能单一的问题，研究提出ALPS预训练视觉状态编码器，在单一patch token集中统一物理状态信息、原生语言对齐及像素重建三大能力。该模型仅使用静态图像与文本从零预训练ViT-B/16，结合语言、重建及空间对应目标，并通过跨视图自蒸馏微调。在涵盖15个公开编码器的统一评测中，ALPS取得ViT-B模型最低的LIBERO状态探测聚合误差，性能匹配3.5倍参数量的V-JEPA 2-L大模型。
在下游任务验证中，冻结的ALPS token配合原生文本塔在LIBERO上达到85.5%的平均策略成功率；在世界模型规划任务（PushT、Wall、Granular）中表现追平或超越主流编码器。技术上，ALPS允许将预测的未来token直接解码为像素图像，实现了从抽象表征到可视化检视的闭环，验证了单一视觉基础同时支撑预测、指令理解与状态检查的可行性，为世界模型提供了通用的视觉预训练范式。
关键看点
- 统一表征架构：在单一patch token集内融合物理状态、语言对齐与像素重建，解决传统编码器能力割裂问题
- 小模型高性能：ViT-B/16规格在LIBERO状态探测误差最低，性能对标3.5倍参数量的V-JEPA 2-L模型
- 策略执行指标：冻结ALPS token搭配原生文本塔，在LIBERO基准测试中实现85.5%平均策略成功率
- 可解释性验证：支持将预测的未来token直接解码还原为像素图像，实现世界模型内部状态的视觉检视
工程与产业观察
ALPS证明了轻量级ViT-B通过多目标联合预训练可兼顾世界模型的预测、控制与可解释性需求。其85.5%的策略成功率与像素级解码能力，为具身智能提供了一种低算力成本下统一感知表征的工程范式，有助于降低端侧部署门槛并提升长程任务调试效率。
英文摘要 (Abstract)
World models turn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection. However, existing encoders typically specialize in only one or two of these capabilities; we therefore introduce ALPS, a pretrained visual state encoder that unifies all three in a single set of patch tokens. Using only still images and text, we pretrain a ViT-B/16 from scratch with language, reconstruction, and spatial correspondence objectives, and then refine it via cross-view self-distillation. In a unified evaluation across fifteen public encoders, ALPS achieves the lowest aggregate LIBERO state-probe error among ViT-B models (matching the 3.5x larger V-JEPA 2-L) while maintaining strong language alignment and accurate pixel reconstruction. Downstream, frozen ALPS tokens reach the highest mean policy success on LIBERO (85.5%) when paired with our native text tower, match or exceed leading encoders on world-model planning across PushT, Wall, and Granular, and allow predicted future tokens to be decoded directly into pixels for visual inspection. We hope this work helps establish visual pretraining as a shared foundation for understanding, prediction, and action in world models.
Hugging Face Daily Papers 热度：11 upvotes 作者：Yanqing Liu, Letian Zhang, Chengzhi Liu, Zijun Wang, Jieru Mei, Hongliang Fei, Yuyin Zhou, Cihang Xie
来源
本页事实对应的公开页面，以原文为准。