外观
论文HF Daily Papers · arXiv 2610.271039 天前
Omni-Woodpecker免训练框架修正全模态LLM跨模态幻觉
本文由 AI 根据公开内容整理，可能有误，请以原文为准：Omni-Woodpecker: Hallucination Correction via Evidence-Prior Rebalancing for Omni LLMs（HF Daily Papers · arXiv 2610.27103）
核心内容
针对全模态大模型（OLLMs）在音视频联合推理中频发的跨模态幻觉问题，研究团队提出免训练框架Omni-Woodpecker（OWP）。该方法摒弃传统输出层干预，转向表征层面的冲突条件重平衡。实验显示，以Qwen2.5-Omni-7B为基座时，OWP在CMM和AVHBench幻觉评测集上分别提升12.9和8.6分，并在四个通用推理基准中获得0.67至2.25分的稳定增益，且成功泛化至VideoLLaMA2-AV架构。
技术上，OWP首先从互补模态视角识别跨模态冲突，利用正交分解将隐藏状态中的接地证据与竞争先验分离；随后自适应抑制误导性先验并恢复被忽略的证据，完成解码前的表征重平衡。该机制无需额外训练即可缓解融合表征中证据与先验的纠缠问题。目前项目代码已在GitHub开源，为具身智能多模态感知系统的可靠性验证提供了新的算法工具与评测参考。
关键看点
- 免训练纠偏：采用冲突条件表征重平衡替代输出层操作，通过正交分解分离隐藏状态中的接地证据与竞争先验，解码前自适应抑制误导信息
- 幻觉评测提升：基于Qwen2.5-Omni-7B骨干网络，在CMM和AVHBench跨模态幻觉基准上性能分别提升12.9分和8.6分，显著优于现有方法
- 跨架构泛化：在四个通用OLLM推理基准上获得0.67-2.25分增益，并成功迁移至VideoLLaMA2-AV模型，CMM与AVHBench分别提升4.6和3.9分
工程与产业观察
该工作揭示了全模态模型内部表征纠缠是幻觉主因，提出的免训练正交分解方案为具身机器人多模态感知系统提供了低成本可靠性增强路径。其开源代码与评测协议可作为VLA模型部署前的安全对齐工具，降低机器人在复杂非结构化环境中因跨模态误判导致的执行风险。
英文摘要 (Abstract)
Recent Omni LLMs (OLLMs) have demonstrated remarkable capabilities in joint audio-visual reasoning, yet cross-modal hallucinations remain a major challenge. Existing mitigation methods mainly operate at the output level, overlooking the entanglement of grounding evidence and learned priors within fused representations. To address this issue, we propose Omni-Woodpecker (OWP), a training-free framework that shifts hallucination mitigation from output-level manipulation to conflict-conditioned representation rebalancing. Specifically, OWP first identifies cross-modal conflicts from complementary modality-specific views. It then employs orthogonal decomposition to separate grounding evidence from competing priors within the hidden states. Finally, OWP adaptively rebalances fused representations by suppressing misleading priors and restoring ignored evidence before decoding. Extensive experiments demonstrate the effectiveness and strong cross-architecture generalization of OWP. Using Qwen2.5-Omni-7B as the backbone, OWP improves performance by 12.9 and 8.6 points on CMM and AVHBench, respectively, while consistently achieving gains of 0.67-2.25 points across four general OLLM reasoning benchmarks. Furthermore, OWP generalizes effectively to VideoLLaMA2-AV, improving performance by 4.6 and 3.9 points on the same benchmarks. Code is publicly available at https://github.com/ZiiJianLiu/Omni-Woodpecker
Hugging Face Daily Papers 热度：21 upvotes 作者：Zijian Liu, Chaoyou Fu, Yan Yang, Caifeng Shan, Ran He
来源
本页事实对应的公开页面，以原文为准。