外观
论文HF Daily Papers · arXiv 2610.278517 天前
LaLa-World语言条件潜世界模型规划提速13.9倍训练降本6倍
本文由 AI 根据公开内容整理，可能有误，请以原文为准：LaLa-World: Scaling Latent World Models with Natural Language Actions（HF Daily Papers · arXiv 2610.27851）
核心内容
针对JEPA式潜世界模型（LWM）跨域学习难题，研究团队发布LaLa-World，采用视频生成模型的语言条件接口，融合仿真、机器人轨迹及通用图像视频数据进行联合训练。实验表明，该模型在MPC规划任务中平均性能超越分域训练的现有LWM，且引入多模态数据显著增强了动作表述鲁棒性。
技术上将数值动作与视觉变化统一为语言描述，构建基于语言模型的未来状态预测器以支持异构数据集联合训练。对比RAEv2视频生成基线，LaLa-World达到同等规划水平但训练成本降低6倍、规划速度快13.9倍。作为预训练底座初始化的LaLa-Policy在LIBERO和RoboCasa上表现对标Cosmos Policy，推理速度最高提升5.5倍。
关键看点
- 跨域统一接口：用自然语言描述数值动作与视觉变化，使单一预测器可在仿真、真机轨迹及通用视频数据上联合训练
- 规划效率跃升：相比RAEv2视频生成基线，LaLa-World实现同等MPC规划性能，训练成本降低6倍、规划推理提速13.9倍
- 策略微调加速：基于LaLa-World初始化的LaLa-Policy在LIBERO和RoboCasa基准上推理速度较Cosmos Policy提升达5.5倍
- 数据增强鲁棒性：引入额外图像和视频数据不仅提升平均规划性能，还显著增强模型对不同动作语言表述的适应能力
工程与产业观察
LaLa-World验证了以语言为统一接口融合多源数据训练潜世界模型的可行性，大幅降低具身规划训练门槛与推理延迟。其“语言即动作”范式为构建通用世界模型提供新路径，对解决真实机器人数据稀缺、提升Sim-to-Real迁移效率具有直接工程参考价值。
英文摘要 (Abstract)
This paper shows that JEPA-style latent world models (LWMs) can learn across diverse domains and data sources by adopting the language-conditioned interface of video generation models. We establish this approach as an effective and efficient basis for planning with imagination and downstream policy fine-tuning. To this end, we introduce LaLa-World, a Language-conditioned Latent World model that learns from simulation and robot trajectories together with broader image and video data. Two designs make this work. First, we describe both numerical actions and visual changes in language, enabling joint training of a single future predictor across heterogeneous datasets. Second, we build the predictor on a language model, allowing it to interpret language conditions while predicting future latent states. The resulting predictor achieves higher average MPC planning performance than existing LWMs trained separately for each domain or task. Adding image and video data further improves average planning performance and robustness to action phrasing. In a matched comparison with an RAEv2-based video generation model baseline, LaLa-World reaches comparable planning performance with 6times less training cost and 13.9times faster planning. Beyond planning, the pretrained LaLa-World serves as an initialization for LaLa-Policy, which jointly predicts actions and future visual representations. LaLa-Policy performs competitively with Cosmos Policy on LIBERO and RoboCasa, with up to 5.5times faster inference.
Hugging Face Daily Papers 热度：37 upvotes 作者：Junha Song, Geonmo Gu, Jeongeun Park, Heejeong Nam, Lucas Maes, Byeongho Heo, Dongyoon Han, Sangdoo Yun
来源
本页事实对应的公开页面，以原文为准。