Paper2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al. · NeurIPS · 2022
Abstract
Speeds up exact attention by attacking memory traffic rather than arithmetic. The observation is that attention is bottlenecked on moving the large intermediate matrix between levels of GPU memory, so the computation is restructured into tiles that stay in fast on-chip memory and the big matrix is never written out at all. The result is identical, not approximated.
Why it matters
The rare optimisation with no accuracy trade-off to weigh, which is why it was absorbed everywhere within a year. It is also the clearest lesson that on modern hardware the bottleneck is usually memory movement, not compute.
attentionefficiencyinfrastructureserving
Read the source
https://arxiv.org/abs/2205.14135