外观
论文arXiv 2610.2124413 天前
System1-Arena：哪些决策可以交给 System 1？
原文链接：System1-Arena: Which Decisions Can Be Left to System 1?（arXiv 2610.21244 · 发布日期：2026-10-15）
System1-Arena: Which Decisions Can Be Left to System 1?
提出 System1-Arena 基准，用统一决策请求评测 System-1 与推理模型，在多类任务和闭环环境中验证。
核心要点
- 动机是区分哪些离散决策可由单次前向的 System-1 模型处理，哪些需要生成式推理。
- 基准用相同决策请求并按概率评分，覆盖 92k 个问题、105 项任务和 4 类任务组。
- 还纳入 138 个闭环环境；实时模式下，世界会在模型决策时继续演化。
- 评测 30 个模型：大型 System-1 非智能体准确率可匹敌或超过快速闭源 LLM，闭环中大型开放 LLM 成功最多。
验证范围与局限：验证范围包括公共来源任务和从网购、SWE-bench 编码控制到机器人、游戏、驾驶的闭环环境。
英文摘要 (Abstract)
Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.
作者：Tianyu Chen, Xinyang Liu, Ruyi An, Zhendong Wang, Mingyuan Zhou
来源
本页事实对应的公开页面，以原文为准。