Paper2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee et al. · J. Mach. Learn. Res. · 2019
Abstract
Recasts every language task — translation, classification, summarisation, question answering — as text in, text out, so a single model and loss covers all of them. On that common footing the authors run a systematic comparison of pretraining objectives, architectures, corpora and scale, isolating which choices actually matter.
Why it matters
Valuable as a controlled study rather than a model release: it is one of the few places where pretraining decisions are compared like-for-like instead of confounded. The text-to-text framing is now simply how everyone uses language models.
pretrainingtransferevalstraining
Read the source
https://arxiv.org/abs/1910.10683