论文/arXiv 2610.20190·14 天前

用于离线强化学习的价值引导排斥

针对漂移策略仅按几何关系排斥动作的问题，VRQL将Q值融入粒子交互场，并在OGBench与D4RL AntMaze配置上验证。

AI 整理

• 具身模型

原文链接：Value-Guided Repulsion for Offline Reinforcement Learning（arXiv 2610.20190 · 发布日期：2026-10-14）

Value-Guided Repulsion for Offline Reinforcement Learning

针对漂移策略仅按几何关系排斥动作的问题，VRQL将Q值融入粒子交互场，并在OGBench与D4RL AntMaze配置上验证。

核心要点

• 标准漂移策略只依据生成动作间的几何关系确定排斥，无法按期望价值区分动作。
• VRQL在每个状态下结合几何核距离与归一化Q值缩放，构建价值引导的交互场。
• 该机制让低价值粒子产生更强排斥，同时严格保持对行为动作的吸引。
• 在50个OGBench任务配置和6个D4RL AntMaze配置上，表现出与既有基线高度竞争的性能。

验证范围与局限：验证范围为50个OGBench任务配置和6个D4RL AntMaze配置。

英文摘要 (Abstract)

Drifting models have emerged as a highly efficient and stable paradigm for policy design in offline reinforcement learning (RL). However, standard drift policies determine repulsion solely from geometric relationships among generated actions and therefore cannot distinguish them by expected value. To address this limitation, we introduce Value-Guided Repulsion Q-Learning (VRQL) to integrate action value estimates into the particle interaction field. For each state, VRQL combines geometric kernel distance with a normalized Q value scaling. This mechanism assigns greater repulsive influence to particles with lower values while strictly retaining attraction toward behavior action. Consequently, this value-guided interaction field aligns behavior regularization with policy improvement without adding an auxiliary behavior model stage. On 50 OGBench task configurations and 6 D4RL AntMaze configurations, VRQL demonstrates highly competitive performance against established baselines, suggesting that value-guided particle interaction is an effective method for offline RL.

作者：Yixue Hao, Jiahao Zhang, Jiesong Lian, Xufei Guo, Xumeng Guo, Wenhui Zhang, Long Hu, Rui Wang

来源

本页事实对应的公开页面，以原文为准。

1. arXiv 2610.20190：Value-Guided Repulsion for Offline Reinforcement Learning访问 2026-10-21