论文arXiv 2609.389662 天前
Video2SwimFish:从视频重建可控鱼类模型与生物运动流形
本文由 AI 根据公开内容整理,可能有误,请以原文为准:Video2SwimFish: An Automated Pipeline for Reconstructing Controllable Fish Models and Biological Locomotion from Real Fish Videos(arXiv 2609.38966)
核心内容
针对水下具身智能仿真资产匮乏问题,研究发布Video2SwimFish自动化管线及基准测试。该方案输入个体鱼的同步多视角视频,通过VLM Actor-Critic循环构建适配形态的内部关节与生物运动流形(BLM)。实验表明,相比直接关节空间学习,基于BLM的低维动作空间使重构个体的游泳策略收敛速度显著提升,任务成功率更高。项目已开源包含6个物种、120条个体的同步顶视与前视视频数据集,以及对应的可控资产与独立游泳策略。
技术核心在于利用VLM从规范帧生成度量级可变形网格,并将真实鱼类运动约束为低维BLM动作空间。评测涵盖3D轨迹跟踪、目标到达及BlueROV机器人捕获仿真案例,并结合专家人工评估验证生物真实性。研究发现任务成功率与运动保真度并非正相关,精准复现个体动物运动仍是开放挑战。该数据集支持基于个体运动忠实度的细粒度评估,为水下机器人Sim-to-Real提供了高保真生物学先验与标准化验证环境。
关键看点
- 生物运动流形:提取BLM作为低维动作空间边界,相比关节空间学习显著提升策略收敛速度与任务成功率
- VLM重建管线:利用VLM Actor-Critic循环从多视角视频自动生成度量级可变形网格与适配形态的内部关节
- 开源数据集:发布6物种120条个体同步双视角视频、可控资产及独立游泳策略,支持个体级运动保真度评测
- 保真度解耦:证实任务成功率与生物运动保真度非正相关,确立个体运动复现为水下具身智能开放挑战
工程与产业观察
该工作将生物运动学先验转化为可计算的流形约束,解决了水下仿生仿真中“形似神不似”的痛点。BLM低维动作空间设计为复杂柔性体控制提供了高效范式,其强调的个体级运动保真度指标,对提升水下机器人Sim-to-Real迁移效率及生物启发算法验证具有重要工程参考价值。
英文摘要 (Abstract)
We present Video2SwimFish, an automated pipeline and benchmark for building controllable fish assets from real-fish videos for underwater embodied AI. Given synchronized multi-view videos of an individual fish, the pipeline generates a metrically scaled deformable mesh from a VLM-selected canonical frame, builds internal articulation adapted to that individual's morphology through a VLM actor-critic loop, and extracts a Biological Locomotion Manifold (BLM). The BLM provides a low-dimensional action space bounded by real-fish motion. Compared with learning directly in joint space, it lets the individual swimming policy of each reconstructed fish converge faster and reach a higher task success rate. We release two paired datasets: synchronized top- and front-view recordings of 120 individual fish across 6 species, and the controllable assets and individual swimming policies derived from them. Because every asset is tied to the animal it came from, the dataset supports a benchmark that evaluates locomotion learning not only on task success but on fidelity to that individual's locomotion behavior. We evaluate on two tasks, 3D trajectory following and target reaching, and a case study in which a simulated BlueROV underwater robot captures the assets. We further validate biological realism with a human study in which fish experts rate the simulated fish, as a complement to the automated metrics. We find that task success and locomotion fidelity do not necessarily improve together, and we identify faithful reproduction of individual animal locomotion as an open challenge for the community. Project website: https://hangongchen.github.io/video2swimfish-web/
作者:Hangong Chen, Linfeng Cheng, Tahsin Zaman Jilan, Jiaye Wu, Lee Caesar, Ian Fuller Yantian Zha
外部来源
本页事实对应的公开页面,以原文为准。(点击将离开本站)