Learn AI · Level 3 · Tech and tools · step 2 of 16 · word 4 of 8
What is LoRA in AI?
LoRA is a cheap way to fine-tune a large model. Instead of changing all its settings, it trains a small add-on that adjusts how the model behaves.
In plain English
Fully fine-tuning a big model means updating billions of parameters, which needs expensive hardware. LoRA freezes the original model and trains a small set of extra numbers alongside it. The add-on is tiny, so you can keep several, one for each task, and swap them in.
An example
Microsoft researchers introduced LoRA in 2021. Compared with fully fine-tuning GPT-3, which has 175 billion parameters, they reported 10,000 times fewer trainable parameters and a third of the GPU memory.
Watch out
LoRA is how most people customise open-weight models and image generators, often on a single gaming graphics card. Variants such as QLoRA cut memory needs further. It adjusts style and skills well, but it is not a good way to add lots of new facts: for that, use RAG.
Learn AI path · Level 3 · Tech and tools · step 2 of 16
Next wordPrompt engineering →See all of level 3