外观
论文arXiv 2609.305541 个月前
面向机器人控制的隐私保护提示式策略搜索
原文链接：Privacy-Preserving Prompted Policy Search for Robotic Control（arXiv 2609.30554 · 发布日期：2026-09-24）
Privacy-Preserving Prompted Policy Search for Robotic Control
PP-ProPS通过客户端秘密变换编码策略与奖励，在多类连续和离散控制问题上验证隐私保护的LLM引导策略搜索。
核心要点
- 动机：云端LLM策略优化需上传原始策略参数和奖励历史，可能暴露专有控制策略。
- 方法：在客户端用秘密变换编码策略参数和奖励，使服务方仅见编码参数和缩放奖励。
- 改进：提供各奖励分量并采用有界历史，避免提示无限增长，支持高维策略和开源权重LLM。
- 实验：在十项任务中七项优于Vanilla ProPS，六项中五项超过PPO、SAC、TRPO。
验证范围与局限：验证范围为摘要所述MuJoCo运动、经典控制、高速公路驾驶和机械臂操作等连续与离散控制问题。
英文摘要 (Abstract)
Large language models (LLMs) have recently demonstrated promising capabilities as in-context policy optimizers for Reinforcement Learning (RL), enabling policy search driven by both numerical reward signals and natural language reasoning. However, deploying such methods in practice requires transmitting raw policy parameters and rewards history to cloud-based LLM APIs, exposing proprietary control strategies to third-party service providers. To address this issue, this paper introduces Privacy-Preserving Prompted Policy Search (PP-ProPS), a framework that enables LLM-guided policy optimization while keeping policy and environmental parameters confidential. PP-ProPS encodes policy parameters and reward values using secret client-side transformations before they are included in each API request, ensuring that the LLM provider observes only encoded policy parameters and scaled reward information. Furthermore, unlike Vanilla ProPS, the proposed framework does not require the true optimal episodic return to be known or disclosed to the LLM. Beyond protecting the optimization data, PP-ProPS improves the search process in two ways. First, it provides the LLM with individual reward components instead of only a single total return, offering more informative feedback about each candidate policy. Second, it uses a bounded history that prevents the prompt from growing indefinitely, improving search with high-dimensional policies and supporting the use of open-weight LLMs. The proposed PP-ProPS is evaluated on both continuous and discrete control problems spanning Multi-Joint dynamics with Contact (MuJoCo) locomotion, classic control, highway driving, and robotic arm manipulation. Compared to Vanilla ProPS, the proposed PP-ProPS outperforms ProPS in seven of the ten evaluated tasks, and surpasses conventional RL methods including PPO, SAC, and TRPO, in five of the six tasks.
作者：Ali Irshayyid, Feng Lin, Chong Li, Jun Chen
来源
本页事实对应的公开页面，以原文为准。