Skip to content

论文arXiv 2611.220774 天前

ReLAP框架:JEPA空间循环预测实现无动作标签预训练 ​

本文由 AI 根据公开内容整理,可能有误,请以原文为准:ReLAP: Latent Action Pretraining through Recurrent Video Prediction in JEPA Space(arXiv 2611.22077)

核心内容 ​

针对机器人行为学习中缺乏动作标注的痛点,arXiv论文提出ReLAP隐式动作预训练框架。该方法在冻结的JEPA表征空间内,通过共享逆动力学模型推断紧凑连续隐式动作,并利用隐式世界模型循环预测未来特征。这种机制使后续预测误差能反向传播至早期动作,强制学习超越成对重建的时序一致性动态。最终策略通过动作流匹配适配到机器人演示数据,部署时移除显式分支由微调VLM直接输出物理动作,显著降低推理复杂度。

实验数据显示,ReLAP仅使用11M预训练样本和1.06B可训练参数,即在标准LIBERO基准上取得97.9%成功率,并在LIBERO-Plus零样本扰动测试中获得70.0%成绩。在LARY隐式动作基准测试中,其平均回归误差与分类准确率均位列第二。技术路径上,模型通过隐式蒸馏与循环未来特征预测双重目标将表征迁移至视觉语言模型,证明了无动作递归预训练可生成紧凑、时序结构化且可迁移的机器人控制表征,为具身智能数据效率优化提供新范式。

关键看点 ​

  • 架构创新:在冻结JEPA空间耦合逆动力学与世界模型,利用循环预测误差反传机制学习时序一致的隐式动作表征
  • 性能指标:1.06B参数模型在LIBERO基准达97.9%成功率,零样本扰动测试70.0%,LARY基准综合排名前列
  • 数据效率:仅依赖11M无动作标注视频样本完成预训练,通过动作流匹配高效适配下游机器人演示任务
  • 部署优化:推理阶段剥离显式隐式动作与世界模型分支,由微调VLM端到端直接预测物理动作以降低计算负载

工程与产业观察 ​

ReLAP验证了在JEPA潜在空间进行循环预测可有效替代显式动作标注,解决了大规模视频数据用于具身智能训练的标签瓶颈。其“预训练学表征、部署端直出动作”的工程解耦思路,兼顾了离线学习的时序一致性与在线推理的轻量化,为构建通用机器人基础模型提供了高数据效率的技术参考。

英文摘要 (Abstract) ​

Videos provide a rich source of physical interaction for learning robot behavior without action annotations, yet effective latent actions must capture not only local transitions but also how those transitions evolve over time. We introduce ReLAP, a latent-action pretraining framework that couples local transition learning with recurrent multi-step prediction in a frozen JEPA representation space. A shared inverse dynamics model infers compact continuous latent actions from adjacent observations, while a latent world model recurrently predicts the future JEPA features. Because each predicted feature becomes the context for the next transition, errors at later steps backpropagate through the rollout to earlier latent actions, encouraging temporally coherent dynamics beyond pairwise reconstruction. We transfer this representation to a vision-language model using two complementary objectives: latent distillation from the frozen inverse teacher and recurrent future-feature prediction through the frozen latent world model. The resulting policy is then adapted to robot demonstrations with action flow matching. At deployment, the explicit latent-action and world-model branches are removed, and the fine-tuned VLM predicts physical actions directly, reducing inference complexity. With only 11M pretraining samples and 1.06B trainable parameters, ReLAP achieves 97.9% on standard LIBERO and 70.0% over zero-shot LIBERO-Plus perturbations. On the LARY latent-action benchmark, ReLAP ranks second in both mean regression error and mean classification accuracy among the compared methods. These results demonstrate that recurrent action-free latent action pretraining can learn compact, temporally structured, and transferable representations for downstream robot control.

作者:R. Gnana Praveen, Ali Rouzbayani, Zhiying Jiang, Bidhan Roy

基于本页技术方向关联的整机参数、核心零部件、实测报告与同赛道选型横评。

外部来源

本页事实对应的公开页面,以原文为准。(点击将离开本站)

  1. arXiv 2611.22077:ReLAP: Latent Action Pretraining through Recurrent Video Prediction in JEPA Space ↗访问 2026-11-18