Paper2021
LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al. · ICLR · 2021
Abstract
Freezes the pretrained weights entirely and trains a pair of small low-rank matrices alongside each one, on the observation that the update a fine-tune applies has far lower rank than the weights themselves. Trainable parameters drop by orders of magnitude, and because the adapters can be folded back into the weights afterwards, inference costs nothing extra.
Why it matters
The reason fine-tuning is something an individual can do on one GPU rather than a datacentre operation. Nearly every 'custom model' offering is this underneath.
fine tuningefficiencysmall modelstraining
Read the source
https://arxiv.org/abs/2106.09685