Multi-Turn On-Policy Distillation with Prefix Replay
Baohao Liao, Hanze Dong, Christof Monz, Xinxing Xu et al. · arXiv · 2026
Abstract
Fully online on-policy distillation for multi-turn agents is expensive, since every update needs fresh rollouts through the environment plus teacher queries at each visited state. This paper names a 'prefix trap': histories closer to the student's own behaviour are more relevant to it, but are exactly where the teacher's guidance is least reliable. The fix reuses pre-collected teacher trajectories as prefixes, sampling with a decaying schedule that favours early positions.
Why it matters
Pairs naturally with the relay-distillation work in this collection — both attack the same tension from opposite sides. Read them together if you are training agents on teacher supervision.
https://arxiv.org/abs/2607.04763