外观
论文arXiv 2610.1979514 天前
StateCraft：用抽象状态推理增强视频世界模型
原文链接：StateCraft: Augmenting Video World Models with Abstract-State Reasoning（arXiv 2610.19795 · 发布日期：2026-10-14）
StateCraft: Augmenting Video World Models with Abstract-State Reasoning
StateCraft将可更新抽象状态与视觉状态解耦，结合语言模型和视频扩散模型，在生成式界面基准和3D概念实验中验证长程一致生成。
核心要点
- 长交互中仅记忆过往观测不足，动作可能改变当前不可见的世界状态。
- 方法维护持久可更新的抽象状态，并用其条件化表示当前视图的视觉状态。
- 构建无人工或智能体监督的数据流水线和基准，衡量生成质量、记忆与可控性。
- 在该基准上超过现有基线，并在长轨迹中保持高自一致性和较强生成质量。
验证范围与局限：验证主要在生成式界面基准上，并包含一个改变未观察房间光照的3D概念验证实验。
英文摘要 (Abstract)
Video world models aim to simulate complex environments with realistic appearance and dynamics. Maintaining consistency over long interactions requires tracking past actions and understanding how those actions affect the environment. Video world models can only generate partial observations of the entire world at a time. Remembering past observations alone may not be sufficient in every scenario, as they may no longer reflect the world after a sequence of actions. We introduce a paradigm that decouples a persistent, updatable abstract state from the visual state. The abstract state keeps track of the entities' changes as the world evolves, and is used to condition the visual state, which represents the current view. We propose StateCraft, a novel architecture that pairs a language model with a video diffusion model to evolve the dynamic world. We evaluate our approach on generative interfaces, which pose a substantial challenge because a single action may change parts of the state that are not currently visible. To support training and evaluation, we develop a data collection pipeline that produces informative, densely annotated trajectories at scale without human or agent supervision, together with a benchmark measuring generation quality, memory, and controllability. On this benchmark, our approach improves surpasses the state-of-the-art baselines. We demonstrate that StateCraft maintains high self-consistency and strong generation quality over long trajectories. Because the abstract state is explicit, it can be edited directly, and generation follows edits to states never observed in the data. Finally, a proof-of-concept 3D experiment, in which actions change the illumination of an unobserved room, shows that the paradigm can also be applied to spatial environments. Project page: https://insait-institute.github.io/StateCraft/.
作者：Lorenzo Venturoli, Nedko Savov, Xi Wang, Marc Pollefeys, Luc Van Gool
来源
本页事实对应的公开页面，以原文为准。