外观
论文arXiv 2403.005042024-03-01
Image World Models扩展JEPA预测全局光度变换表征
本文由 AI 根据公开内容整理，可能有误，请以原文为准：Learning and Leveraging World Models in Visual Representation Learning（arXiv 2403.00504）
核心内容
针对联合嵌入预测架构（JEPA）仅局限于输入缺失部分预测的问题，研究提出Image World Models（IWM）方法。该方法突破掩码图像建模范式，在潜空间中学习预测全局光度变换效应。论文系统研究了高性能IWM的训练策略，并验证了通过微调该预测式世界模型可解决多样化下游任务，其性能匹配或超越此前主流自监督学习方法。
IWM技术方案核心在于重构预测任务，使其涵盖更广泛的图像损坏类型而非单纯掩码。研究明确指出构建高效IWM依赖三大要素：条件化机制、预测难度设定及模型容量配置。此外，该方法赋予表征学习可控性，既能习得类似对比学习的不变性表征，也能获取类似掩码建模的等变性表征，为具身视觉感知提供了灵活的预训练基座。
关键看点
- 任务泛化：IWM将JEPA预测目标从局部掩码扩展至潜空间全局光度变换，突破传统掩码图像建模局限
- 训练三要素：确立条件化、预测难度与模型容量为构建高性能图像世界模型的关键技术配方
- 表征可控性：支持调节抽象层级，按需学习不变性或等变性表征，适配不同具身感知任务需求
- 下游性能：微调后的IWM世界模型在多项任务中匹配或超越现有自监督方法，验证预测范式有效性
工程与产业观察
IWM证明了在潜空间预测物理变换比单纯重建像素更能学到鲁棒表征。对具身智能而言，这种可控的不变/等变特性直接关联Sim-to-Real迁移效率，为机器人在光照剧变等非结构化环境下的视觉编码器设计提供了优于传统MAE的理论支撑与工程路径。
英文摘要 (Abstract)
Joint-Embedding Predictive Architecture (JEPA) has emerged as a promising self-supervised approach that learns by leveraging a world model. While previously limited to predicting missing parts of an input, we explore how to generalize the JEPA prediction task to a broader set of corruptions. We introduce Image World Models, an approach that goes beyond masked image modeling and learns to predict the effect of global photometric transformations in latent space. We study the recipe of learning performant IWMs and show that it relies on three key aspects: conditioning, prediction difficulty, and capacity. Additionally, we show that the predictive world model learned by IWM can be adapted through finetuning to solve diverse tasks; a fine-tuned IWM world model matches or surpasses the performance of previous self-supervised methods. Finally, we show that learning with an IWM allows one to control the abstraction level of the learned representations, learning invariant representations such as contrastive methods, or equivariant representations such as masked image modelling.
作者：Quentin Garrido, Mahmoud Assran, Nicolas Ballas, Adrien Bardes, Laurent Najman, Yann LeCun
来源
本页事实对应的公开页面，以原文为准。