Paper2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova · NAACL-HLT · 2018
Abstract
Introduces bidirectional pretraining for language: instead of predicting the next word, mask words at random and predict them from context on both sides. That objective lets every layer condition on the full sentence, and the pretrained encoder can then be fine-tuned for a specific task with a single added output layer.
Why it matters
The paper that made 'pretrain once, fine-tune cheaply' the default. Encoder models like this still underpin most retrieval and classification stacks, including the reranking step in a typical RAG pipeline.
architecturepretrainingattentiontransfer
Read the source
https://arxiv.org/abs/1810.04805