Agentic Reinforcement Learning

SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

Completed on-policy experience becomes evolving hindsight supervision.

Jinyang Wu1,*,† Shuo Yang1,* Zhengxi Lu2 Fan Zhang3 Yuhao Shen2 Lang Feng4 Zheng Lian5 Shuai Zhang1 Zhengqi Wen1 Jianhua Tao1
1Tsinghua University 2Zhejiang University 3The Chinese University of Hong Kong 4Nanyang Technological University 5Tongji University

* Equal contribution    † Project leader

HF Paper Code Model Contact
On-Policy Distillation Dense Token Supervision Self-Evolving Hindsight Skills
Overall performance with Qwen2.5-3B-Instruct. SEED achieves the strongest average performance on ALFWorld and WebShop while remaining competitive on Search-based QA without using skill prompts during evaluation.

Overview

Closing the supervision gap in long-horizon agentic learning

Outcome rewards tell an agent whether an episode succeeded. SEED teaches it which decisions its completed experience supports.

Abstract

Large language models are increasingly trained as interactive agents for long-horizon tasks involving multi-turn interaction, tool use, and environment feedback. Outcome-based reinforcement learning provides a practical optimization paradigm, but sparse trajectory-level rewards offer limited guidance on intermediate decisions.

SEED converts completed on-policy trajectories into natural-language hindsight skills and distills their behavioral effect back into the policy. The current policy both collects trajectories and analyzes them, so decision making and hindsight supervision evolve together. Skill-conditioned rescoring turns probability shifts on sampled actions into dense token-level supervision. The resulting signal is jointly optimized with outcome-based reinforcement learning.

01

On-policy

Guidance is extracted from states, actions, and failures induced by the current policy.

02

Dense

Trajectory-level hindsight is converted into supervision for individual sampled action tokens.

03

Self-evolving

The shared actor and analyzer improve together as reinforcement learning updates the policy.

Method

One policy, two synchronized roles

The latest policy acts in the environment and analyzes its completed trajectories. OPD internalizes the behavioral effect of the resulting hindsight skills.

  1. 1
    CollectSample grouped on-policy trajectories.
  2. 2
    AnalyzeExtract reusable hindsight skills with the synchronized policy.
  3. 3
    Re-scoreEvaluate sampled actions with and without the skill context.
  4. 4
    DistillOptimize dense OPD supervision jointly with outcome-based RL.
SEED framework. Stage 1 initializes trajectory analysis through hindsight-skill supervised fine-tuning. Stage 2 repeatedly synchronizes the actor and analyzer, then combines outcome-based RL with on-policy distillation.

Results

Stronger learning from every completed trajectory

Across three backbones and diverse agentic domains, SEED improves performance, sample efficiency, and transfer to unseen settings.

91.8 ALFWorld average Qwen2.5-3B
60% Training data beats full-data GRPO
+15.3 Unseen average gain over GRPO on ALFWorld
91.0 Vision-task average versus 77.0 for GRPO
A

Dense hindsight improves credit assignment

The gains over GRPO show that token-level hindsight provides more effective credit assignment than terminal rewards alone.

B

Internalized skills outperform inserted prompts

SEED requires no skill context at inference time and still outperforms skill-prompting baselines on nearly all aggregate comparisons.

C

Evolving supervision beats static distillation

Synchronizing the analyzer with the latest policy adapts hindsight to newly encountered trajectories and failure modes.

Multimodal extension

Hindsight supervision also transfers through vision

Using Qwen2.5-VL-3B-Instruct, SEED extends to Sokoban spatial planning and EZPoints visual arithmetic. The extracted skills are used only during training, so evaluation requires no skill prompt.

Sokoban82.0+14.9 over GRPO
EZPoints100.0+13.1 over GRPO
Average91.0+14.0 over GRPO
A representative six-step Sokoban trajectory requiring visual state tracking and long-horizon spatial planning.

Efficiency and transfer

Efficient learning and transfer to unseen tasks

Sample efficiency. With only 60% of the training data, SEED reaches 80.7 and surpasses GRPO trained on the full dataset at 75.0.
Cross-domain generalization. SEED raises the ALFWorld Unseen average from 70.9 to 86.2 and improves five of six task families.
Training dynamics on ALFWorld. SEED learns faster and reaches higher success. Shorter episodes coincide with better performance, indicating more direct task completion.

Qualitative evidence

From unproductive exploration to direct completion

On the same ALFWorld task, GRPO enters an off-task loop. SEED searches plausible locations, finds the target object, and completes the task in five steps.

Qualitative comparison on ALFWorld. The SEED policy preserves the task goal, checks plausible shelves, and executes the correct final placement without receiving a skill prompt.

Citation

Resources and citation

Code, training recipes, and the released checkpoint are available through the project resources.

GitHub Hugging Face
BibTeX 
@article{wu2026seed,
  title  = {SEED: Self-Evolving On-Policy Distillation for
            Agentic Reinforcement Learning},
  author = {Wu, Jinyang and Yang, Shuo and Lu, Zhengxi and
            Shen, Yuhao and Zhang, Fan and Feng, Lang and
            Zhang, Shuai and Luo, Haoran and Lian, Zheng and
            Wen, Zhengqi and Tao, Jianhua},
  year   = {2026},
  note   = {Preprint}
}