Whitepaper2023
Mistral 7B
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford et al. · Mistral AI (Technical Report) · 2023
Abstract
A seven-billion-parameter model that outperformed considerably larger contemporaries, built on two attention changes: grouped-query attention, which shrinks the memory the cache consumes during generation, and sliding-window attention, which bounds how far back each layer looks so long inputs cost linearly rather than quadratically.
Why it matters
The paper that made small open models credible. Both attention tricks are now near-universal, so it doubles as the clearest short explanation of why modern models serve cheaply.
architectureattentionefficiencyopen weights
Read the source
https://arxiv.org/abs/2310.06825