外观
论文arXiv 2609.3751929 天前
Video2STL：VLM生成时序逻辑规范实现跨具身机器人策略学习
本文由 AI 根据公开内容整理，可能有误，请以原文为准：Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning（arXiv 2609.37519）
核心内容
针对视频策略学习中时序结构难以复用与跨具身迁移难题，研究团队提出Video2STL框架。该方法利用视觉语言模型从纯观察视频中提取与具身无关的语义事件轨迹，构建符号化时序规范库，并通过成功机器人轨迹锚定数值阈值与时序边界。在四项机械臂操作任务评测中，Video2STL取得85.8%的平均单次成功率和67.0%的终态成功率，分别比原生Dense PPO高出4.3和7.5个百分点，比Text2Reward高出20.8和24.7个百分点，验证了形式化时序表征在复杂任务学习中的有效性。
技术架构上，Video2STL解耦短时与长时时序信息：短视域规范通过滚动窗口定量鲁棒性提供密集奖励，长视域规范经因果监控器对有效时序前缀给予一次性进度奖励。该表征支持从人或动物视频到机器人控制的跨具身迁移。在四足运动任务中，基于Qwen-3.8和GPT-5.6生成的Video2STL策略在0.3至2.1 m/s全速度区间实现100%成功率，且高速能效保持竞争力。项目代码与文档已在video2stl.github.io开源，为无动作标注视频驱动机器人学习提供了可检验、可复用的新范式。
关键看点
- 时序逻辑表征：将视频转为参数化STL规范，分离短/长时域奖励机制，解决时序结构不可检问题
- 操作任务性能：4项任务平均Success-once达85.8%，Success-at-end达67.0%，全面超越PPO与Text2Reward
- 跨具身迁移：支持从人/动物观察视频提取语义事件轨迹，无需动作标注即可迁移至机器人控制策略
- 四足运动验证：Qwen-3.8/GPT-5.6驱动策略在0.3-2.1m/s全速段100%成功，高速能效具竞争力
工程与产业观察
Video2STL以信号时序逻辑替代标量奖励或代码生成，使视频到策略的映射具备可解释性与跨具身复用能力。其短/长时域解耦奖励设计解决了稀疏反馈难题，在操作与运动任务中均验证了形式化规范对提升样本效率与泛化性的工程价值，为无标注视频数据驱动的具身智能训练提供了可扩展技术路径。
英文摘要 (Abstract)
Video-based policy learning is particularly promising, as it illustrates target behaviors without requiring action annotations or embodiment-matched demonstrations. A central challenge is deciding what information should be transferred from the video to the robot. Existing approaches commonly convert visual observations into scalar similarity or value signals, or ask foundation models to directly generate reward code. These approaches can make the temporal structure of a task difficult to inspect, ground, and reuse. We present Video2STL, a framework that converts observation-only videos into parametric Signal Temporal Logic (STL) specifications and uses the resulting formal representation for robot learning. A vision-language model extracts an embodiment-independent semantic event trace and constructs a bank of symbolic temporal specifications. The model determines the task structure, while numerical predicate thresholds and temporal bounds are grounded from successful robot trajectories. For policy learning, we separate short- and long-timescale temporal information: short-horizon specifications provide dense rewards through rolling-window quantitative robustness, while a causal monitor over a retained long-horizon specification provides one-time progress rewards for valid temporal prefixes. The same representation supports cross-embodiment transfer from human or animal videos to robot control. Across four manipulation tasks, Video2STL achieves $85.8%$ average success-once and $67.0%$ success-at-end, compared with $81.5%/59.5%$ for native dense PPO and $65.0%/42.3%$ for Text2Reward; in quadruped locomotion, Qwen-3.8 and GPT-5.6-based Video2STL policies achieve $100%$ success across velocities from $0.3$ to $2.1,\mathrm{m/s}$ while remaining competitive in high-speed energy efficiency. Project webpage: \href{https://video2stl.github.io/}{video2stl}.
作者：Merve Atasever, Keyan Azbijari, Cagan Bakirci, Bo-Ruei Huang, Tolga Izdas, Zahra Shahrooei, Richard Yang, Erdem Biyik, Jyotirmoy V. Deshmukh
来源
本页事实对应的公开页面，以原文为准。