外观
论文arXiv 2606.123664 个月前
APT预训练框架解决VLA指令泛化难题，兼容GR00T架构
本文由 AI 根据公开内容整理，可能有误，请以原文为准：APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies（arXiv 2606.12366）
核心内容
针对Vision-Language-Action（VLA）模型在分布外（OOD）语言指令上泛化能力弱的痛点，arXiv论文提出APT（Action Expert Pretraining）框架。研究指出VLA数据中语言多样性远低于视觉与动作，导致连续动作专家从随机初始化学习时产生噪声梯度，破坏VLM语义能力。APT从贝叶斯视角将策略分解为语言无关的视觉-动作（VA）先验和语言条件的VLA似然，通过两阶段训练解决结构性失衡，在未见指令及组合任务上实现一致性性能提升。
技术实现上，Stage 1在冻结VLM参数下，仅利用视觉-动作对预训练动作专家作为VA先验，规避语言数据 imbalance；Stage 2引入门控融合机制（Gated Fusion）注入语言Token，在整合VLM特征的同时保留已学习的视运动先验。该方法不依赖特定模型结构，明确验证了对主流π架构及NVIDIA GR00T风格架构的兼容性，为具身智能基础模型的指令跟随训练提供了通用的数据效率优化范式。
关键看点
- 贝叶斯策略分解：将VLA策略因子化为语言无关的VA先验与语言条件似然，从数学层面解决多模态数据结构性失衡导致的梯度噪声问题
- 两阶段预训练机制：Stage 1冻结VLM仅用视觉-动作对训练动作专家，Stage 2通过门控融合注入语言，防止语言短板腐蚀视觉表征
- 架构通用性验证：方法适配主流VLA范式，实验证实兼容π架构及GR00T风格架构，在OOD指令与组合操作任务上取得稳定增益
- 根因分析定位：揭示连续动作专家因缺乏离散动作的视觉-语言共训保护，在数据不平衡下易陷入视觉捷径，导致语言理解失效
工程与产业观察
APT揭示了具身大模型训练中“动作专家冷启动”与“多模态数据失衡”的工程矛盾。其解耦预训练思路为GR00T等架构提供了低成本提升指令泛化的路径，表明在高质量语言-动作对齐数据稀缺现状下，通过训练范式创新挖掘现有VLM潜力比单纯扩大数据规模更具工程性价比。
英文摘要 (Abstract)
Vision-Language-Action (VLA) models that couple pretrained Vision-Language Models (VLMs) with continuous action experts have achieved strong manipulation performance, yet generalization to out-of-distribution (OOD) language instructions remains poor. A known challenge is the structural imbalance in VLA data, where language is far less diverse than visual and action content, making policies prone to visual shortcuts. While discrete-action methods mitigate this through vision-language co-training, continuous action experts lack such protection: they start from random initialization and learn entirely from imbalanced data, producing noisy gradients that corrupt the VLM and fail to exploit its language capability. We address this from a Bayesian perspective, factorizing the policy into a language-agnostic Vision-Action (VA) prior and a language-conditioned VLA likelihood, and propose APT, a two-stage training method emphasizing Action expert PreTraining. In Stage 1, the action expert is pretrained as a VA prior on vision-action pairs from a frozen VLM, bypassing the language imbalance. In Stage 2, language tokens are injected through a gated fusion mechanism that integrates VLM features while preserving the learned visuomotor prior. APT applies to mainstream VLA architectures, including the $π$ and GR00T-style architectures. Comprehensive experiments validate that APT achieves consistent gains on unseen instructions and compositional tasks. Project Page: https://xukechun.github.io/papers/APT/
作者：Kechun Xu, Zhenjie Zhu, Anzhe Chen, Rong Xiong, Yue Wang
来源
本页事实对应的公开页面，以原文为准。