论文HF Daily Papers · arXiv 2610.39264昨天

在线密集RL框架GiGPPO实现iOS UI导航8B模型性能超越Gemini

研究提出解耦规划与UI定位的在线强化学习框架，利用LLM-as-a-Judge提供密集奖励。8B参数模型在iOS真机任务中追平Gemini-3.1-Pro，Android跨域泛化能力提升7倍，验证了端侧具身智能体高效训练路径。

AI 整理

仿真与训练
具身模型
端侧部署

本文由 AI 根据公开内容整理，可能有误，请以原文为准：Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live IOS Apps（HF Daily Papers · arXiv 2610.39264）

核心内容

针对前沿VLM推理成本高及小模型SFT训练脆弱的问题，研究团队提出一种直接在实时iOS应用上训练的在线强化学习框架。该方案将智能体解耦为Planning Agent与UI Grounding Model，通过LLM-as-a-Judge提供步骤级密集反馈与终端奖励双重信号。评测显示，8B参数RL模型在真实iOS应用中性能追平Gemini-3.1-Pro，并在Android环境跨域泛化测试中较SFT基线提升近7倍，显著优于仅使用终端奖励的RL基线。

技术层面采用PPO算法结合步骤级优势估计优化策略，并创新提出GiGPPO邻居条件价值估计法，利用并发轨迹中视觉相似状态的奖励信号增强Critic稳定性。为支撑大规模在线训练，构建了可编排数百台并发虚拟iOS设备的基础设施。在UINavBench、iOSWorld及AndroidWorld等基准测试中，该方法不仅加速了训练收敛，还证明了密集奖励机制对于解决UI导航长序列决策与错误恢复问题的有效性，为移动端具身智能提供了低成本高性能范式。

关键看点

架构解耦：将UI导航智能体拆分为Planning Agent与UI Grounding Model，配合LLM-as-a-Judge提供步骤级密集奖励与终端奖励，解决小模型记忆静态轨迹导致的执行脆弱问题。
算法创新：提出GiGPPO邻居条件价值估计方法，利用并发轨迹中视觉相似状态奖励信号优化Critic稳定性，结合PPO步骤级优势估计，显著提升在线RL训练收敛速度。
性能突破：8B参数RL模型在真实iOS应用表现追平Gemini-3.1-Pro，Android跨域泛化能力较SFT基线提升近7倍，验证了在线密集RL在端侧UI导航任务中的有效性。
训练基建：构建支持数百台虚拟iOS设备并发编排的高效基础设施，支撑大规模在线RL训练范式，解决了实时应用交互环境下的数据获取与策略迭代瓶颈。

工程与产业观察

该研究证明了在移动端具身场景中，通过架构解耦与密集奖励机制，8B级小模型可逼近超大模型性能。GiGPPO算法与并发仿真基建为机器人UI自动化及端侧Agent训练提供了可复用的工程范式，对降低具身智能落地成本具有重要参考价值。

英文摘要 (Abstract)

While frontier vision-language models demonstrate strong capabilities in user interface (UI) navigation, their high inference costs prohibit everyday use. Conversely, smaller models trained via supervised fine-tuning (SFT) or offline reinforcement learning often memorize static trajectories, leading to brittle execution and an inability to recover from errors. To address these limitations, we propose a novel online reinforcement learning framework that trains UI navigation agents directly on live iOS applications. Our framework decouples the agent into two specialized components: a Planning agent and a UI Grounding model. We rely on an LLM-as-a-Judge to provide comprehensive reward signals, combining dense, step-level feedback to evaluate intermediate progress with a terminal reward for overall task completion. To effectively leverage this dual reward structure, we optimize the agent using Proximal Policy Optimization (PPO) with step-level advantage estimation. Furthermore, we introduce GiGPPO, a novel neighbor-conditioned value estimation method that improves critic stability by leveraging reward signals from visually similar states across concurrent trajectories. To support this online training paradigm at scale, we introduce an efficient infrastructure that orchestrates hundreds of concurrent virtual iOS devices. Evaluations on rigorous benchmarks, including UINavBench, iOSWorld, and AndroidWorld, demonstrate that our online RL trained agents substantially outperform both SFT and terminal-only RL baselines while accelerating training convergence. Notably, our 8B RL agents achieve nearly a sevenfold improvement over the SFT baseline in out-of-distribution generalization to Android environments, while outperforming Gemini-3.5-Flash and tying with Gemini-3.1-Pro on real iOS applications.

Hugging Face Daily Papers 热度：3 upvotes 作者：Peitong Duan, Khashayar Gatmiry, Jeffrey Nichols, Samy Bengio, Emmanuel Abbe