Paper2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon et al. · NeurIPS · 2023
Abstract
Removes the reward model and the reinforcement-learning loop from preference tuning. The authors show the standard objective can be rearranged so the language model itself implicitly represents the reward, which means preferences can be optimised with an ordinary classification loss on pairs of responses — no sampling, no separate critic, no reward-hacking to police.
Why it matters
It made preference tuning something a small team can run, which is why most open fine-tunes now use this rather than the reinforcement-learning pipeline it replaced. A good example of a derivation, not a bigger model, being the contribution.
post trainingreinforcement learningtrainingrubrics
Read the source
https://arxiv.org/abs/2305.18290