Paper2024
OLMo: Accelerating the Science of Language Models
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia et al. · ACL · 2024
Abstract
An open language model released with the things that usually stay private: the full pretraining corpus, the data-preparation code, the training logs, hundreds of intermediate checkpoints, and the evaluation harness. The model itself is mid-sized; the contribution is that every stage between raw text and released weights can be inspected and rerun.
Why it matters
'Open-weights' and 'open' are not the same claim, and this is the paper that makes the distinction concrete. If you want to study how a model acquires a capability during training rather than just probe the finished artefact, this is the one that lets you.
open weightspretrainingtrainingopen source
Read the source
https://arxiv.org/abs/2402.00838