论文/arXiv 2604.04539·3 个月前

FlashSAC：面向高维机器人控制的快速稳定离策略强化学习

针对高维机器人控制中离策略RL收敛慢且不稳定，FlashSAC以少更新、大模型和范数约束，在多仿真任务及人形运动Sim-to-Real中验证。

AI 整理

• 本体 SDK 与控制
• 仿真与训练
• 数据与采集

原文链接：FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control（arXiv 2604.04539 · 发布日期：2026-07-13）

FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control

针对高维机器人控制中离策略RL收敛慢且不稳定，FlashSAC以少更新、大模型和范数约束，在多仿真任务及人形运动Sim-to-Real中验证。

核心要点

• 高维状态和动作空间中，同策略方法数据分布窄，离策略方法又易收敛慢且不稳定。
• FlashSAC基于Soft Actor-Critic，减少梯度更新，并以更大模型和更高数据吞吐补偿。
• 方法显式约束权重、特征和梯度范数，以在扩展规模时抑制critic误差累积。
• 在10个模拟器60余项任务中优于PPO和强离策略基线，人形运动Sim-to-Real训练由小时降至分钟。

验证范围与局限：验证范围包括10个模拟器的60余项任务，以及Sim-to-Real人形运动；真机数量和具体任务细节摘要未说明。

英文摘要 (Abstract)

Reinforcement learning (RL) is a core approach for robot control when expert demonstrations are unavailable. On-policy methods such as Proximal Policy Optimization (PPO) are widely used for their stability, but their reliance on narrowly distributed on-policy data limits accurate policy evaluation in high-dimensional state and action spaces. Off-policy methods can overcome this limitation by learning from a broader state-action distribution, yet suffer from slow convergence and instability, as fitting a value function over diverse data requires many gradient updates, causing critic errors to accumulate through bootstrapping. We present FlashSAC, a fast and stable off-policy RL algorithm built on Soft Actor-Critic. Motivated by scaling laws observed in supervised learning, FlashSAC sharply reduces gradient updates while compensating with larger models and higher data throughput. To maintain stability at increased scale, FlashSAC explicitly bounds weight, feature, and gradient norms, curbing critic error accumulation. Across over 60 tasks in 10 simulators, FlashSAC consistently outperforms PPO and strong off-policy baselines in both final performance and training efficiency, with the largest gains on high-dimensional tasks such as dexterous manipulation. In sim-to-real humanoid locomotion, FlashSAC reduces training time from hours to minutes, demonstrating the promise of off-policy RL for sim-to-real transfer.

作者：Donghu Kim, Yongkyu Lee, Minho Park, Kinam Kim, Takuma Seno, I. Nahrendra, Sehee Min, Daniel Palenicek, FLORIAN M. VOGT, Danica Kragić

来源

本页事实对应的公开页面，以原文为准。

1. arXiv 2604.04539：FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control访问 2026-10-21