论文HF Daily Papers · arXiv 2611.009118 天前
S1CogBench发布:首个System One决策模型认知基准评测
本文由 AI 根据公开内容整理,可能有误,请以原文为准:S1CogBench: What System One Decision Models Can and Cannot Do on Cognitive Experiments(HF Daily Papers · arXiv 2611.00911)
核心内容
针对具身智能所需的快速直觉决策能力,研究团队发布S1CogBench基准,涵盖24项经典认知实验与27种评估模式,对7个开源模型及Jev API进行系统测试。该基准结合人类反应预测、自主执行与控制变量操作,旨在解决传统聚合评分掩盖模型认知缺陷的问题,首次为System One决策模型建立了标准化认知能力评估体系。
评测发现模型存在显著行为分化:部分模型虽在特定学习任务上准确率高,但在动态环境中缺乏信息搜集与反馈适应能力,且高分不等于类人响应。技术层面证实读取头(readout)设计影响排名,延迟对比显示直接决策比长文本生成快数个数量级,而匹配的单token生成几乎无额外开销,验证了System One架构在低延迟具身控制中的工程可行性。
关键看点
- 基准规模:集成24项认知任务与27种评估模式,覆盖人类反应预测与自主执行双维度
- 认知缺陷:模型高准确率与反馈适应能力解耦,动态环境下信息搜集能力普遍不足
- 延迟特性:直接决策显著快于生成式语言模型,单token输出几乎不增加推理耗时
- 评估偏差:共享读取头控件实验表明,模型性能排名受选择提取方式显著影响
工程与产业观察
S1CogBench填补了具身智能System One快速决策模型的认知评测空白。其揭示的“高分低能”现象警示开发者不能仅凭静态指标选型;低延迟单token决策特性则验证了该类模型在实时机器人控制回路中的部署潜力,建议将其作为具身决策算法准入测试标准。
英文摘要 (Abstract)
Decision-making in humans and AI involves multiple cognitive abilities. System One decision models produce fast, structured choices. Their cognitive strengths and limitations, however, remain poorly understood. We introduce S1CogBench, a benchmark of 24 established cognitive tasks and 27 evaluation modes, evaluating seven open models and the API model Jev. It combines human-response prediction, autonomous execution, and controlled task manipulations to reveal strengths and failures that aggregate scores miss. We find distinct behavioral profiles across the evaluated models. In these models, strong performance on some learning tasks coexists with failures to gather information or adapt to feedback, while high accuracy does not guarantee human-like responses to changing task demands. Shared readout controls show that model rankings depend partly on how choices are extracted. Separate latency comparisons with generative language models show that long responses are much slower than direct decisions, whereas matched one-token generation adds little overhead. To our knowledge, S1CogBench is the first benchmark of System One decision models on established cognitive experiments. It provides a common empirical basis for identifying their capabilities and limitations before integrating them into automated decision systems.
Hugging Face Daily Papers 热度:1 upvotes 作者:Dengzhe Hou, Kazunori D Yamada
外部来源
本页事实对应的公开页面,以原文为准。(点击将离开本站)