外观
论文arXiv 2609.297601 个月前
PolyUMI：面向物体推断与操作的可访问视觉-触觉-音频数据采集
原文链接：PolyUMI: Accessible Visual-Tactile-Audio Data Collection for Object Inference and Manipulation（arXiv 2609.29760 · 发布日期：2026-09-24）
PolyUMI: Accessible Visual-Tactile-Audio Data Collection for Object Inference and Manipulation
PolyUMI通过开源无线手持夹爪采集同步视觉、触觉、音频与本体数据，并用VisTA在物体推断、滑移控制和富接触操作中验证。
核心要点
- 现有模仿学习多依赖视觉和本体感受，难以获取视觉不易推断的接触信息。
- PolyUMI是开源平台，无线手持夹爪同步记录腕部相机、光学触觉、接触音频和本体观测。
- 同一传感手指可转移到机器人末端，VisTA在传感器和时间上融合多模态token以预测动作。
- 实验显示触觉和音频提供超越视觉的任务信息，VisTA可媲美或优于现有多模态策略。
验证范围与局限：验证范围涵盖物体推断、滑移控制和富接触操作任务。
英文摘要 (Abstract)
Humans typically rely on vision, touch, hearing, and proprioception to perceive contact and adapt their actions during manipulation. Providing robots with comparable responsiveness therefore requires hardware that can retain and use these complementary sensory signals. Most imitation-learning systems, however, observe demonstrations primarily through vision and proprioception, limiting access to contact information that is difficult to infer visually. We present PolyUMI, an open-source platform for scalable visual--tactile--audio demonstration collection and robot deployment. Its lightweight, wireless handheld gripper records synchronized wrist-camera, optical tactile, contact-audio, and proprioceptive observations without requiring a tethered workstation. The same sensing finger can be transferred to the robot end effector, preserving the sensing geometry between demonstration collection and policy execution. To effectively use these heterogeneous observations, we further introduce VisTA, a token-level multimodal policy that integrates information across sensors and time to predict contact-aware robot actions. Experiments spanning object inference, slip control, and contact-rich manipulation show that touch and audio reveal task-relevant information beyond vision and that VisTA is competitive with or outperforms existing multimodal policies. Together, PolyUMI and VisTA provide an accessible pipeline for collecting multimodal demonstrations and learning policies that perceive physical interaction beyond vision. Project Page: https://polyumi-vista.github.io
作者：Conor W. Hayes, Rickmer Krohn, Aravind Ramaswami, Anunth Ramaswami, Nils Dengler, Kevin M. Lynch, Ed Colgate, Georgia Chalvatzaki, Matthew L. Elwin
来源
本页事实对应的公开页面，以原文为准。