外观
论文HF Daily Papers · arXiv 2610.1194220 天前
LGWM模型以表征空间预测实现GUI智能体17ms低延迟安全验证
本文由 AI 根据公开内容整理，可能有误，请以原文为准：Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents（HF Daily Papers · arXiv 2610.11942）
核心内容
针对GUI智能体仅检测屏幕内容易受合法界面重放攻击的问题，研究提出LGWM（Latent GUI World Model）作为状态转换验证器。该模型摒弃文本或图像生成范式，采用无解码器架构直接在观测编码空间进行动作条件预测。基于185万条真实GUI转换数据训练，无需语义标注即可学习状态转移逻辑，将安全验证转化为高效的向量相似度计算问题。
在RSWT-BENCH诊断基准测试中，LGWM实现了0.987的AUC评分，单次决策延迟仅17ms，性能对标顶级闭源VLM，但比生成式GUI世界模型快三个数量级以上。其残差方向分析达到0.953 AUC，能有效区分恶意劫持与良性违规，而提示驱动的VLM在此任务上接近随机水平。这表明在表征空间预测不仅可作为模拟器，更是具身智能体实时安全监控的最优设计路径。
关键看点
- 表征空间预测：LGWM采用无解码器架构直接预测下一屏幕表征向量，避免生成文本/图像再二次判断的冗余，将验证简化为向量比对。
- 极致推理效率：单次安全决策耗时仅17ms，比生成式GUI世界模型快三个数量级，满足具身智能体高频交互下的实时安全监控需求。
- 高精度转换验证：在RSWT-BENCH上AUC达0.987，残差方向区分有害/良性违规模型AUC为0.953，显著优于仅看屏幕的检测器及提示VLM。
- 大规模无监督训练：利用185万条真实GUI转换数据进行无语义标注训练，证明预测信号是可用未来状态而非单纯异常分数。
工程与产业观察
LGWM验证了“表征空间预测”在具身智能安全监控中的工程价值。相比高延迟的生成式VLM，直接在潜空间建模状态转换能以毫秒级延迟实现高精度验证。这为机器人GUI操作、端侧部署及安全护栏提供了新范式：安全验证不必依赖昂贵的语义理解，低维向量比对即可高效拦截状态劫持攻击。
英文摘要 (Abstract)
A login screen that appears after a tap on Sign in is expected; the same screen after a tap on View order is an attack. For GUI agents, safety is therefore a property of the transition rather than of the screen, and a monitor that inspects only screens can be defeated by reusing a legitimate one. Judging a transition requires an expectation of what should have followed the action. Existing GUI world models provide one, but they output it as text, code, or images, so checking it against the observed screen requires a second model to judge the two. We argue that a world model meant for verification should instead predict in the space in which observations are encoded, and present LGWM, a decoder-free, action-conditioned world model that predicts the representation of the next screen directly, trained without semantic annotation on 1.85M real GUI transitions. Verification reduces to a vector comparison, and the same signal reveals whether a mismatch is harmful. We evaluate on RSWT-BENCH, a diagnostic where each credential screen appears under both a legitimate and a hijacked transition, so detectors that see only the screen are at chance by construction. The training-free score reaches 0.987 AUC at 17 ms per decision, on par with the strongest closed-source VLMs and about ten AUC points above generative GUI world models at over three orders of magnitude lower latency. The residual direction reaches 0.953 AUC at separating harmful from benign violations, where prompted VLMs are near chance. Further analyses show that the prediction is a usable future state rather than an anomaly score. World models have mostly served as simulators or planners; our results point to a third role, verification, for which predicting in representation space is the natural design.
Hugging Face Daily Papers 热度：81 upvotes 作者：Jiaming Zhang, Xuan Wang, fffyzhang, Yang Cao, Lingjuan Lyu, Wei Yang Bryan Lim
来源
本页事实对应的公开页面，以原文为准。