论文/arXiv 2610.05180·24 天前

VICON：面向操作数据集的基于视觉-惯性-接触的手-物体跟踪

VICON用视觉惯性手套、RGB-D与接触传感联合跟踪手物体运动和接触，并在40次动捕会话中验证。

AI 整理

• 本体 SDK 与控制
• 数据与采集
• 评测与安全

原文链接：VICON: Visual-Inertial-Contact based Hand-Object Tracking for Manipulation Datasets（arXiv 2610.05180 · 发布日期：2026-10-04）

VICON: Visual-Inertial-Contact based Hand-Object Tracking for Manipulation Datasets

VICON用视觉惯性手套、RGB-D与接触传感联合跟踪手物体运动和接触，并在40次动捕会话中验证。

核心要点

• 灵巧操作学习需要接触点与力监督，单靠运动轨迹难以完整捕获交互信息。
• 框架采用视觉惯性手套和RGB-D相机，并在手套上布置FSR记录接触与法向力。
• 物体位姿不依赖预设CAD模型，因子图融合可见性、FSR测量和手运动先验。
• 在40次会话五个物体上失败帧率为2.5%，并将公开发布覆盖10类物体的扩展数据集。

验证范围与局限：验证范围为40次动捕会话和五个物体，包含手-物体遮挡条件。

英文摘要 (Abstract)

Learning dexterous manipulation benefits from human demonstration datasets that capture diverse and natural hand-object interactions. In particular, contact points and forces provide supervision on where and how strongly to interact, which cannot be fully captured by motion trajectories alone. However, methods for jointly capturing hand and object motion, contact points, and forces remain limited. Moreover, severe occlusion from hand-object interaction challenges accurate tracking of both hands and objects. To address these limitations, we present a Visual-Inertial-CONtact based hand-object tracking (VICON) framework. It holistically captures both hand and object motion along with contact information during manipulation, even under severe occlusion. First, we adopt a visual-inertial glove and an RGB-D camera for accurate hand tracking, and redesign the glove to incorporate contact sensing. Specifically, force-sensitive resistors (FSRs) are placed on the glove based on human grasp frequency to synchronously record contact states and calibrated normal forces. Second, without requiring pre-existing CAD models, we estimate object poses using RGB-D images and a mesh reconstructed from a monocular video. We propose factor-graph-based object trajectory estimation that fuses object-pose estimates weighted by visibility under hand-object occlusion, FSR measurements, and a hand-motion prior. Across 40 motion-capture sessions with five objects, VICON achieves a 2.5% failed-frame rate compared with 50.9-64.6% for the baselines, with median errors of 3.9 mm and 3.0 degrees under occlusion. Using VICON, we construct a dataset containing synchronized hand-object motion, contact points, and normal forces, and will publicly release an expanded dataset covering 10 object categories at https://github.com/VICON-dataset/dataset.

作者：Yubin Jeon, Uiseong Shin, Hwanchul La, Jaeseong Kang, Hyelim Choi, 이용석

来源

本页事实对应的公开页面，以原文为准。

1. arXiv 2610.05180：VICON: Visual-Inertial-Contact based Hand-Object Tracking for Manipulation Datasets访问 2026-10-21