Skip to content

论文HF Daily Papers · arXiv 2611.214796 天前

LynnReal-Hand发布:手控视频世界模型实现540p@18FPS实时交互生成 ​

本文由 AI 根据公开内容整理,可能有误,请以原文为准:LynnReal-Hand: Unlocking World Interaction with Hand Control(HF Daily Papers · arXiv 2611.21479)

核心内容 ​

针对具身智能中手物交互视觉反馈生成的挑战,研究团队发布LynnReal-Hand手控视频世界模型。该工作构建了包含10万条真实与合成环境交互片段的数据集,并开发手部感知监督机制以优化细粒度动作建模。在HandBench评测中,全序列模型取得FVD 181.67与MANO Dice 93.02的领先指标,三步因果模型MANO Dice仍保持92.49,验证了其在复杂交互动态下的高保真生成能力。

为满足实时性需求,模型引入因果流图实现少步生成,并结合区域感知分数蒸馏融合视频与图像先验。核心创新Ultra-Fast-VAE紧凑解码器大幅降低计算开销,使系统在单张H200 GPU上实现540p分辨率18.03 FPS流式输出,端到端延迟仅741ms,峰值显存占用控制在34.89GB。团队将开源完整数据集、模型权重及训练推理代码,支撑手控世界模型研究。

关键看点 ​

  • 交互数据规模:构建10万条涵盖真实与合成场景的手物交互视频片段,配合手部感知监督机制提升细粒度动作与物体状态一致性
  • 生成质量指标:全序列模型FVD达181.67、MANO Dice为93.02;三步因果模型MANO Dice维持92.49,平衡精度与推理效率
  • 实时推理性能:基于Ultra-Fast-VAE解码器,单H200实现540p@18.03 FPS流式生成,延迟741ms,峰值显存仅34.89GB
  • 开源资产承诺:将释放完整交互数据集、模型权重及全流程训练推理代码,降低手控视频世界模型研究门槛

工程与产业观察 ​

该工作将手控视频世界模型从离线生成推向实时交互阶段,18 FPS低延迟与34.89GB显存占用使其具备端侧部署潜力。10万条手物交互数据与开源代码为具身智能视觉仿真提供关键基础设施,加速Sim-to-Real中精细操作策略的验证闭环。

英文摘要 (Abstract) ​

Interactive video world models enable exploration through text and camera controls, yet direct object interaction requires finer control. Hands provide a natural interface, but translating continuous hand movements into coherent visual responses remains challenging because of complex hand-object dynamics and demanding real-time computation. To address these challenges, we introduce LynnReal-Hand, a hand-controlled video world model that unifies precise interaction modeling with efficient streaming generation. We curate 100K interaction clips spanning diverse real and synthetic environments and develop hand-aware supervision to improve fine-grained hand motions and object interactions. Building on this foundation, we learn a causal flow map for few-step generation and introduce region-aware score distillation that combines complementary video and image priors to refine interaction dynamics and visual appearance. To further reduce inference latency, we propose Ultra-Fast-VAE, a compact video decoder that preserves high reconstruction fidelity with substantially lower computational cost. Extensive evaluations on HandBench demonstrate superior hand-following accuracy and visual quality compared with existing approaches. Our full-sequence model achieves an FVD of 181.67 and a MANO Dice score of 93.02, while the three-step causal model maintains a MANO Dice score of 92.49. For real-time interaction, 540p streaming on a single H200 reaches 18.03 FPS with 741ms latency and an optimized peak GPU memory footprint of only 34.89GB. We will release the datasets, model weights, and the full training and inference code to support research on hand-controlled video world models.

Hugging Face Daily Papers 热度:1 upvotes 作者:Shaohao Rui, Chaoyang Zhang, Xiaofeng Mao, Weijie Ma

基于本页技术方向关联的整机参数、核心零部件、实测报告与同赛道选型横评。

外部来源

本页事实对应的公开页面,以原文为准。(点击将离开本站)

  1. HF Daily Papers · arXiv 2611.21479:LynnReal-Hand: Unlocking World Interaction with Hand Control ↗访问 2026-11-18