论文/arXiv 2610.09254·21 天前

RoboRender：用于视觉 Sim-to-Real 迁移的面向机器人视频生成

RoboRender将仿真轨迹生成逼真RGB视频用于策略学习，并在视频测试集和真实机器人任务中验证零样本迁移。

AI 整理

• 本体 SDK 与控制
• 感知与导航
• 仿真与训练

原文链接：RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer（arXiv 2610.09254 · 发布日期：2026-10-07）

RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer

RoboRender将仿真轨迹生成逼真RGB视频用于策略学习，并在视频测试集和真实机器人任务中验证零样本迁移。

核心要点

• 仿真训练策略常因视觉差异难以迁移，现有中间表示可能丢失语义或需额外感知模块。
• 方法以仿真深度视频、语言指令和机器人RGB掩码为条件，生成保留几何与动作标签的逼真视频。
• 生成视频与仿真状态、动作配对训练策略，真实任务平均成功率达71%，优于两类基线。
• 生成质量优于深度条件视频生成基线；项目网站已在摘要中给出。

验证范围与局限：验证范围为机器人视频测试集，以及真实世界抓放、关节物体操作和移动操作任务。

英文摘要 (Abstract)

Simulation enables large-scale, low-cost robot data generation, but policies trained in simulation often fail to transfer to the real world due to the sim-to-real visual discrepancies. Existing approaches often rely on intermediate representations, which can discard rich semantic information or require additional perception modules at deployment. We address this visual sim-to-real gap with RoboRender, a framework that converts simulated trajectories into photorealistic RGB videos for policy learning. RoboRender trains a robot-oriented video generation model conditioned on simulated depth videos, language instructions, and robot RGB mask videos, preserving simulator geometry, robot motion, and action labels while synthesizing realistic textures, backgrounds, and distractors. The generated RGB videos are paired with simulator-provided states and actions to train policies for zero-shot real-world deployment. On robot video test sets, our video model outperforms depth-conditioned video generation baselines in generation quality. In real-world experiments across pick-and-place, articulated-object manipulation, and mobile manipulation tasks, policies trained on RoboRender-generated data achieve a 71% average success rate, outperforming raw simulation renderings and conventional visual domain randomization by approximately 7.1x and 3.6x, respectively. We further show that policy performance improves with more generated videos per simulation trajectory, increasing opening-task success by 65 percentage points. These results demonstrate that generative video rendering mitigates the visual sim-to-real gap for zero-shot policy transfer. Project website: https://robo-render.github.io/.

作者：Huang Huang, Wensi Ai, Ziyu Chen, Youhui Wang, Zijian Du, Yang Liu, Jiaolong Yang, Li Fei-Fei, Jiajun Wu

来源

本页事实对应的公开页面，以原文为准。

1. arXiv 2610.09254：RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer访问 2026-10-21