外观
论文arXiv 2510.137273 个月前
从拒绝到恢复：生成式 AI 护栏的控制理论方法
原文链接：From Refusal to Recovery: A Control-Theoretic Approach to Generative AI Guardrails（arXiv 2510.13727 · 发布日期：2026-07-15）
From Refusal to Recovery: A Control-Theoretic Approach to Generative AI Guardrails
该文将智能体 AI 安全视为序贯决策问题，用安全关键控制与强化学习构建模型无关预测护栏，并在模拟驾驶和电商中验证。
核心要点
- 现有护栏多依赖输出分类和拒绝执行，面对新危险情境脆弱且缺少恢复路径。
- 方法在 AI 模型的潜在世界表征中引入安全关键控制，实时监测动作并纠正风险输出。
- 提出可扩展训练方案，通过安全关键强化学习计算可包裹任意 AI 模型的护栏。
- 模拟驾驶和电商实验显示，该护栏可引导 LLM agents 远离碰撞、破产等灾难结果并保持任务性能。
验证范围与局限：验证范围为模拟驾驶和电商设置中的 LLM agents。
英文摘要 (Abstract)
Generative AI systems are increasingly assisting and acting on behalf of end users in practical settings, from digital shopping assistants to next-generation autonomous cars. In this context, safety is no longer about blocking harmful content, but about preempting downstream hazards like financial or physical harm. Yet, most AI guardrails continue to rely on output classification based on labeled datasets and human-specified criteria, making them brittle to new hazardous situations. Even when unsafe conditions are flagged, this detection offers no path to recovery: typically, the AI system simply refuses to act—which is not always a safe choice. In this work, we argue that agentic AI safety is fundamentally a sequential decision problem: harmful outcomes arise from the AI system’s continually evolving interactions and their downstream consequences on the world. We formalize this through the lens of safety-critical control theory, but within the AI model’s latent representation of the world. This enables us to build predictive guardrails that (i) monitor an AI system’s outputs (actions) in real time and (ii) proactively correct risky outputs to safe ones, all in a model-agnostic manner so the same guardrail can be wrapped around any AI model. We also offer a practical training recipe for computing such guardrails at scale via safety-critical reinforcement learning. Our experiments in simulated driving and e-commerce settings demonstrate that control-theoretic guardrails can reliably steer LLM agents clear of catastrophic outcomes (from collisions to bankruptcy) while preserving task performance, offering a principled dynamic alternative to today’s flag-and-block guardrails.
作者：Ravi Pandya, Madison Bland, Duy Phong Nguyen, Changliu Liu, Jaime F. Fisac, Andrea V. Bajcsy
来源
本页事实对应的公开页面，以原文为准。