外观
论文arXiv 2609.288381 个月前
不确定性门控探索噪声抑制Flow-Matching视觉-语言-动作策略在线RL微调中的任务坍塌
原文链接：Uncertainty-Gated Exploration Noise Suppresses Task Collapse in Online RL Fine-Tuning of a Flow-Matching Vision-Language-Action Policy（arXiv 2609.28838 · 发布日期：2026-09-23）
Uncertainty-Gated Exploration Noise Suppresses Task Collapse in Online RL Fine-Tuning of a Flow-Matching Vision-Language-Action Policy
研究SmolVLA在LIBERO-10在线PPO微调中的任务坍塌，提出不确定性门控探索噪声并对比验证。
核心要点
- 在线微调预训练Flow-Matching VLA时，整体表现可能掩盖单任务能力被破坏。
- 方法用任务无关的新颖性和能力信号，在无任务标签和回合边界下重分配探索。
- 固定噪声两种子坍塌，学习噪声每种子坍塌；控制器三种子均未坍塌但未超BC基线。
- 作者发布按四种定义测量逐任务坍塌、重评分噪声和instrument tares的工具。
验证范围与局限：验证限于LIBERO-10上小计算预算的450M参数SmolVLA在线PPO微调，且未解耦状态与时间自适应贡献。
英文摘要 (Abstract)
Online reinforcement learning fine-tuning of pretrained flow-matching vision-language-action (VLA) policies promises robots that keep learning after deployment, but continued updates often destroy competence on individual tasks while the aggregate still looks healthy. We study this failure mode, which we call task collapse, under a matched small-compute budget on LIBERO-10 with a 450M-parameter SmolVLA policy trained by PPO with stochastic (SDE) sampling. Three exploration-noise policies differ in one live variable: a fixed noise scale, a ReinFlow-style learned noise network, and an uncertainty-gated controller that redistributes exploration across task streams from task-agnostic novelty and competence signals, without task labels or episode boundaries. Under the pooled definition, fixed noise collapses tasks in two of three seeds and learned noise in every seed measured to iteration 200, while the controller collapses none in any of its three seeds. Measured parameter displacement shows the controller's action expert keeps changing, while its mean applied noise is close to the fixed scale in the available logs. The matched comparison supports the controller's effect on task preservation; the separate contributions of its adaptation across states and over time are not disentangled. A lower fixed scale slows the decline but does not stop it. No arm improves on the behavior-cloning baseline in this budget. Two properties of that regime are measured beside this result, not offered as its cause: following the reference recipe, training runs in bfloat16 with no fp32 master copy, under which 96.02% of the action expert's elements stay bit-identical across three consecutive iterations, and an fp32 master copy at the reference learning rate collapses both arms in a single-seed observation. We release tools measuring per-task collapse under four definitions, rescoring noise and instrument tares.
作者：Mehmet Turan Yardımcı, Yunus Emre Cogurcu
来源
本页事实对应的公开页面，以原文为准。