论文HF Daily Papers · arXiv 2611.203217 天前
SPEED+基准揭示仿真误设致视觉导航置信度虚高与迁移不对称
本文由 AI 根据公开内容整理,可能有误,请以原文为准:Unreasonable Confidence from Simulator Misspecification: Directional Robustness on the SPEED+ Sim-to-Real Benchmark(HF Daily Papers · arXiv 2611.20321)
核心内容
针对太空视觉导航缺乏真实标注数据的痛点,研究利用SPEED+基准探究仿真器辐射度量误设(缺失镜面反射、硬阴影及传感器饱和)引发的Sim-to-Real差距。通过单目位姿估计器的2x2对照实验发现,模型存在严重的“不合理自信”:在合成数据上校准至90%的共形预测集,在真实Sunlamp和Lightbox域仅分别覆盖9-15%和21-38%的旋转误差,导致错误位姿携带了正确预测的高置信度,构成系统级安全隐患。
实验证实Sim-to-Real迁移具有显著方向不对称性。单纯扩大合成数据规模虽将合成域旋转误差从1.65 rad降至1.03 rad,但在强光照Sunlamp域仅改善0.05 rad;相比之下,适配骨干网络末层可将63-83%的合成域增益有效迁移至真实域。此外,光度增强仅在骨干冻结时对Sunlamp有效,适配后仅对漫反射Lightbox域有效(误差由1.17降至0.99 rad)。相关代码、逐图预测结果及预注册记录已在GitHub开源。
关键看点
- 置信度失效:合成域90%校准置信度在真实Sunlamp/Lightbox域仅覆盖9-15%/21-38%旋转误差,错误预测呈现虚假高置信。
- 迁移不对称性:数据扩容使合成误差降0.62 rad但强光照域仅降0.05 rad;骨干适配可迁移63-83%额外增益至真实域。
- 增强策略局限:光度增强仅在骨干冻结时改善强光照域;骨干适配后仅对漫反射域有效,误差从1.17 rad降至0.99 rad。
- 开源资产:完整实验代码、逐图预测数据及预注册文档已在GitHub开源,支持复现与Sim-to-Real鲁棒性验证。
工程与产业观察
该研究为具身智能Sim-to-Real提供了关键工程警示:仿真器物理误设会导致模型置信度校准失效,使安全阈值形同虚设。算法工程师应摒弃单纯堆叠合成数据的路线,优先采用骨干网络适配策略以获取跨域增益,并针对特定光照条件设计定向增强。开源的SPEED+基准与预注册实验范式,可作为评估视觉导航系统真实可靠性的标准参考。
英文摘要 (Abstract)
Vision-based navigation systems for orbit must be trained almost entirely on synthetic imagery, because real labelled images of a given spacecraft in flight rarely exist. Benchmarks like SPEED+ are built for that situation: the training set comes from a simulator, and the test sets are real images of a satellite mockup under two kinds of lighting. That simulator approximates how light reflects off mylar and aluminium, and like every scientific simulator it is misspecified. Its error is radiometric, since it misses the specular highlights, hard shadows and sensor saturation common in real images. We probe the resulting synthetic-to-real gap, the central risk for such systems, with a deliberately simple monocular pose estimator. In a 2x2 study of backbone adaptation and photometric augmentation, trained on synthetic images and tested on both real hardware-in-the-loop domains, we find three things. (1) The model does not know it is failing: regions conformally calibrated to 90% on synthetic data cover only 9-15% of rotation errors on sunlamp and 21-38% on lightbox, so a wrong pose carries the confidence of a correct one. (2) Transfer is asymmetric: scaling data and features improved synthetic rotation error from 1.65 to 1.03 rad but moved the harshly lit sunlamp domain by only 0.05 rad, under twice its seed s.d., whereas adapting the backbone's last stage carried 63-83% of its extra synthetic gain to both real domains. (3) Once the backbone adapts, augmentation helps only in the direction it models: our photometric stack improved the diffusely lit lightbox domain from 1.17 to 0.99 rad on every seed but moved sunlamp no more than seed-to-seed variation, while with the backbone frozen it helped sunlamp more. Every post-review experiment was pre-registered, and the code, per-image predictions and pre-registration record are at https://github.com/lokindoo/spacecraft-pose-estimation.
Hugging Face Daily Papers 热度:1 upvotes 作者:Niko
外部来源
本页事实对应的公开页面,以原文为准。(点击将离开本站)