外观
论文HF Daily Papers · arXiv 2610.311626 天前
HiSTEVE：分层VLA架构实现Minecraft原生键鼠操控与长程任务
本文由 AI 根据公开内容整理，可能有误，请以原文为准：HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games（HF Daily Papers · arXiv 2610.31162）
核心内容
针对开放世界游戏中具身智能体缺乏长程自主性与原生接口控制协同的问题，研究团队提出HiSTEVE分层VLA架构。该系统在MineStudio Bench基准测试中取得85.0%的原子任务性能，在OpenHA评测中平均成功率达37.4%，并成功完成从零制作铁镐等类人长程操作任务，验证了VLA在非API环境下的端到端执行能力。
技术层面采用双层架构：编排层利用结构化记忆与VLM规划器选择子任务，引入策略耦合状态检测器判定完成度；控制层使用单一VLA执行纯视觉感知与原生键鼠动作。为消除GUI与非GUI交互干扰，设计了特定域RMSNorm分支及独立鼠标移动Token嵌入，并通过世界知识注入、模仿学习与强化学习反馈联合训练提升策略鲁棒性。
关键看点
- 分层架构：编排层集成结构化记忆与VLM规划器，控制层VLA直接输出原生键鼠动作，解耦长程规划与底层执行
- 状态检测：提出策略耦合任务完成状态检测器，基于VLA策略上下文信号判定子任务完成，替代传统环境API反馈
- 交互优化：引入Regime-specific RMSNorm分支与独立鼠标移动Token嵌入，有效缓解异构GUI与非GUI操作的特征干扰
- 量化性能：MineStudio Bench原子任务得分85.0%，OpenHA平均成功率37.4%，支持纯视觉驱动的铁镐制作全流程
工程与产业观察
该工作验证了VLA在脱离环境API条件下通过原生人机接口执行长程任务的可行性。其策略耦合状态检测与异构交互归一化设计，为解决具身智能在非标数字环境中的感知-控制对齐问题提供了工程范式，对仿真训练向真实GUI操作迁移具有参考价值。
英文摘要 (Abstract)
Building virtual embodied agents that operate in open-ended games like human players is a long-standing goal requiring both long-horizon autonomy and reliable native-interface control. As a procedurally generated open world, Minecraft provides a natural testbed for studying the integration of these capabilities. Existing agents typically provide either native visuomotor control without long-horizon autonomy or high-level planning executed through environment APIs or predefined skills. To bridge this gap, we propose HiSTEVE, a hierarchical VLA agent with a capability-aware agent harness. At the orchestration level, we equip the harness with structured memory for tracking progress and a VLM planner for selecting subtasks within the VLA's capabilities. To reliably determine subtask completion, we further introduce a policy-coupled task completion state detector that conditions on VLA policy context to signal candidate completion. At the control level, a single VLA executes all subtasks with vision-only perception and native keyboard-and-mouse actions. To mitigate interference between heterogeneous GUI and non-GUI interactions, we introduce regime-specific RMSNorm branches and separate mouse-movement token embeddings. We also train the VLA through world knowledge injection, imitation learning, and reinforcement learning with environment feedback to improve reliability and recovery. HiSTEVE achieves state-of-the-art atomic-task performance, scoring 85.0% on MineStudio Bench and a 37.4% average success rate on OpenHA, while also completing long-horizon tasks such as crafting an iron pickaxe from scratch like human players.
Hugging Face Daily Papers 热度：31 upvotes 作者：Zecong Tang, Zian Guan, Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Bo Dai
来源
本页事实对应的公开页面，以原文为准。