AI Engineering · interactive
LoRA: fine-tune a sliver of the model
Full fine-tuning changes every weight and needs a whole new copy of the model. LoRA freezes the original weights and trains two thin matrices beside them. Their product is the same shape as the weights it adjusts, at a fraction of the size.
A tiny 8×8 layer at rank 2 (the picture stops at 2)
Trained here: 32 numbers (B and A) to adjust all 64 weights, and the update can only change the layer in 2 independent directions. The saving grows with size: at 4096×4096, rank 8 trains 0.39% as many numbers.
Full fine-tune of those layers (a 7B model)1.07B
LoRA at rank 84.2M
Rank 8: train 4.2M numbers instead of 1.07B, 0.06% of the model. The adapter file is 8 MB.
1 2 3 Gradients flow through the frozen model but only update the adapters, so training fits on far smaller GPUs.
4.2M trainable numbers, 0.39% of a full fine-tune of the same layers.
4
In production
Tools and technologies
LoRA is the default way to fine-tune open models, and many hosted APIs fine-tune with adapters too.
Not tied to one language — use these names when you search docs or design a system.
- Hugging Face PEFTSoftware
LoRA, QLoRA and other adapters for Transformers models.
GitHub ↗ - UnslothSoftware
Faster, lower-memory LoRA fine-tuning on one GPU.
GitHub ↗ - vLLM multi-LoRASoftware
Serve one base model with many adapters loaded per request.
Official site ↗ - LoRA paperStandard
Hu et al., 2021: Low-Rank Adaptation of Large Language Models.
Official site ↗