外观
论文HF Daily Papers · arXiv 2610.2334412 天前
LM2LM利用研究世界模型实现LLM递归自改进与低成本RL训练
本文由 AI 根据公开内容整理，可能有误，请以原文为准：Language Models Learn to Improve Language Models（HF Daily Papers · arXiv 2610.23344）
核心内容
针对大模型自动化研发中强化学习反馈获取昂贵及代理反馈失真痛点，研究提出LM2LM框架。该方法利用冻结的研究世界模型（RWM）预测提案收益作为RL奖励，并通过周期性执行高分提案校准上下文。实测显示，Qwen3.8-27B在8个自回归与扩散模型开发环境中，性能超越获300条真实记录加持的GPT-6-sol，且数据生成成本降低33%，并具备跨模型族泛化能力。
技术核心在于构建可扩展且可靠的反馈闭环：RWM基于真实实验记录预测增益，替代全量训练评估以规避数十万GPU-hours消耗；为防止策略优化导致分布偏移下的奖励过估，系统定期执行当前策略的高排名提案并将实测结果回注RWM上下文。该架构将昂贵的模型训练反馈转化为低成本的上下文推理预测，为具身智能算法及基础模型的自动化迭代提供了标准化的低成本RL训练范式。
关键看点
- RWM奖励机制：采用冻结LLM作为研究世界模型，基于上下文真实实验记录预测提案增益，替代全量训练评估以解决RL反馈获取昂贵问题
- 动态校准策略：周期性执行当前策略下RWM高排名提案并将实测结果回注上下文，有效缓解分布偏移导致的代理反馈过估与可靠性下降
- 性能与成本优势：Qwen3.8-27B在8个LM开发环境中超越GPT-6-sol，同时数据生成成本降低33%，验证了低成本递归自改进的有效性
- 跨族泛化能力：LM2LM框架在未参与训练的模型家族中仍保持有效，证明基于RWM的RL优化策略具备通用性而非过拟合特定架构
工程与产业观察
该研究确立了“世界模型+动态校准”作为AI研发自动化的标准范式。对具身智能而言，这意味着可利用仿真或历史数据构建RWM，以极低算力成本训练策略生成器，再辅以少量真机/高精仿真数据校准，破解具身算法迭代中“评估贵、样本少”的工程瓶颈，加速Sim-to-Real与模型自适应进化。
英文摘要 (Abstract)
Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training.
Hugging Face Daily Papers 热度：5 upvotes 作者：Minhua Lin, Zijun Wang, Zewen Liu, Zhaotian Weng, Bing He, Zhan Shi, Benoit Dumoulin, Xiang Zhang, Suhang Wang, Hanqing Lu
来源
本页事实对应的公开页面，以原文为准。