具身开发者社区

外观

论文arXiv 2610.1785615 天前

MMTA：基于时间序列、文本与视觉的多模态时序分析基准测试 ​

为评估MLLM整合时间序列与真实视觉证据，MMTA以共享模式和加权指标构建三模态任务，并在九领域测试集验证。

AI 整理
  • 具身模型
  • 端侧部署
  • 评测与安全

原文链接：MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision（arXiv 2610.17856 · 发布日期：2026-10-13）

MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision

为评估MLLM整合时间序列与真实视觉证据，MMTA以共享模式和加权指标构建三模态任务，并在九领域测试集验证。

核心要点 ​

  • 现有基准多将时序信号配文本或同源可视化，难检验与真实视觉证据的融合能力。
  • MMTA覆盖时间序列、文本、视觉，包含跨模态互补模式，并统一异构真实来源与领域提示。
  • 基准跨九个领域，含video+TS和image+TS布局，3524个测试样本覆盖分类与预测。
  • 评估八个MLLM发现文本化时序导致长token和视觉细粒度对齐弱；TimeOmni-v提升长视频分类和时序推理效率。

验证范围与局限：验证范围为九领域video+TS和image+TS布局的分类与预测测试集，并评估八个开源和专有MLLM。

英文摘要 (Abstract) ​

Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations derived from the same signal. This leaves open whether multimodal large language models (MLLMs) can integrate time series with real-world visual evidence. We introduce MMTA (multimodal temporal analysis), a benchmark of tri-modal tasks with diverse cross-modal complementarity patterns among time series, text, and vision. MMTA spans nine domains -- robotics, autonomous driving, human action, manufacturing, AR assistance, healthcare, recommendation, finance, and climate -- under video+TS and image+TS layouts, with a 3,524-sample test split covering classification and forecasting. The benchmark standardizes heterogeneous real-world sources into a shared schema with domain-specific prompts, provenance documentation, and success-weighted metrics that penalize both wrong answers and failed generations. Evaluating eight open-source and proprietary MLLMs reveals a shared bottleneck: representing time series as text creates very long token sequences and weakens fine-grained alignment with visual evidence, especially in long-video settings. We then introduce TimeOmni-v, a multimodal LLM extension with a dedicated time-series encoder, temporal positional alignment, modality-aware token layout, and a forecasting head. TimeOmni-v improves long-video classification and time-series inference efficiency, showing that MMTA not only exposes representation and grounding failures but also motivates targeted model design.

作者：Ziyang Zhang, Shenyi Li, Yilin Wang, Ziyun Cui, Bowen Zhou, Wen Wu, Chao Zhang

来源

本页事实对应的公开页面，以原文为准。

  1. arXiv 2610.17856：MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision访问 2026-10-21