DeepSeek-V3 Technical Report
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue et al. · DeepSeek (Technical Report) · 2024
Abstract
The engineering report behind a 671-billion-parameter mixture-of-experts model that activates only a fraction of itself per token. It is unusually candid about the parts papers normally omit: the load-balancing strategy that keeps experts evenly used without an auxiliary loss fighting the main objective, the mixed-precision training that made the run affordable, and the communication scheduling that keeps thousands of GPUs busy.
Why it matters
Read it for the training economics rather than the benchmarks. It is the clearest public account of how a frontier-scale run is actually made affordable, and much of what followed in open-weight modelling traces back to choices documented here.
https://arxiv.org/abs/2412.19437