Fine-tuning without touching the model
LoRA (Low-Rank Adaptation) is a technique for fine-tuning a large model with far less computation and data. Rather than modifying the model’s own enormous parameter set, it trains a small difference and attaches it to the base.
How to picture it
Ordinary fine-tuning retunes the whole model, which needs many GPUs and a lot of time. LoRA freezes the base and trains only a small adapter. The effect approaches full fine-tuning at a fraction of the cost — sticking notes onto a book rather than rewriting it.
The resulting LoRA file is small, which makes it easy to distribute and swap.
Where it is used
- Image generation: a large ecosystem of LoRAs for reproducing particular styles, characters, or people in Stable Diffusion
- LLMs: the standard way to fine-tune a local model on your own GPU
- Business use: lightweight tuning on internal data, keeping costs down
It broke the assumption that fine-tuning was only for large organizations and opened model customization to individuals.
Cautions
LoRAs that reproduce a specific artist’s style or a real person invite copyright and personality-rights problems. The technique is neutral, but what you train it on requires thought. When an article mentions training a LoRA or a style LoRA, read it as a small add-on trained on top of a base model.
The name refers to expressing the difference in a reduced-rank matrix. The mathematics matters less than knowing it as the standard term for cheap, fast fine-tuning.