具身开发者社区

外观

论文arXiv 2610.2187013 天前

通过优化确定性等价（OCE）实现可证明高效的风险敏感离线强化学习 ​

本文用OCE统一刻画风险度量，设计悲观值迭代/最小二乘值迭代，在表格与线性函数逼近设置下给出离线风险敏感RL理论保证。

AI 整理
  • 数据与采集
  • 具身模型
  • 端侧部署

原文链接：Provably Efficient Risk-Sensitive Offline Reinforcement Learning via Optimized Certainty Equivalents（arXiv 2610.21870 · 发布日期：2026-10-15）

Provably Efficient Risk-Sensitive Offline Reinforcement Learning via Optimized Certainty Equivalents

本文用OCE统一刻画风险度量，设计悲观值迭代/最小二乘值迭代，在表格与线性函数逼近设置下给出离线风险敏感RL理论保证。

核心要点 ​

  • 固定数据学习风险敏感策略很重要，但离线场景的一般理论仍不足。
  • 研究动态OCE与静态OCE两种形式，分别递归或一次性作用于奖励。
  • 在表格设置提出悲观值迭代，并建立紧的次线性次优界和极小极大下界。
  • 面向大状态空间，给出结合OCE的悲观最小二乘值迭代及依赖d的保证。

验证范围与局限：验证范围为理论分析的表格设置和线性函数逼近，摘要未报告实验或真机验证。

英文摘要 (Abstract) ​

Risk-sensitive reinforcement learning (RL) addresses decision-making under uncertainty and adverse outcomes. Learning risk-sensitive policies from fixed datasets is important when additional interaction with the environment is costly. Existing work on risk-sensitive RL focuses on online settings or specific risk measures, leaving a general theoretical understanding of its offline setting underdeveloped. In this work, we investigate efficient offline risk-sensitive RL using the optimized certainty equivalent (OCE), a general class of risk measures. We consider two formulations: dynamic-OCE RL, which applies OCE recursively at each step, and static-OCE RL, which applies a single OCE to the cumulative rewards. Under the tabular setting, we develop pessimistic value-iteration algorithms for both formulations and establish tight sublinear suboptimality bounds, explicitly capturing how risk sensitivity, encoded by utility functions, affects learning difficulty. Moreover, we establish minimax suboptimality lower bounds to characterize the problem's fundamental limits. To handle large state spaces, we further develop pessimistic least-squares value-iteration algorithms that incorporate the general OCE risk criterion into linear function approximation. We establish utility-dependent sublinear guarantees with explicit dependence on the feature dimension $d$. To our knowledge, we provide the first offline algorithms along with the first upper and lower suboptimality bounds for both dynamic- and static-OCE RL.

作者：Hanzhen Wang, Boxiang Lyu, Zhongxiang Dai, Siyang Gao, Dake Zhang, Shuang Qiu

来源

本页事实对应的公开页面，以原文为准。

  1. arXiv 2610.21870：Provably Efficient Risk-Sensitive Offline Reinforcement Learning via Optimized Certainty Equivalents访问 2026-10-21