LoRA Learns Less and Forgets Less

Author(s): Hesam Sheikh

Originally published on Towards AI.

We will go through LoRA (Low-Rank Adaptation of Large Language Models), what it is, and the interesting properties of LoRA when compared to Full Fine-Tuning
LoRA from the original paper.

LoRA is one of the ABCs of working with LLMs. To intuitively know what LoRA does, why it is important in fine-tuning, and, most importantly, when not to use LoRA is an essential topic to know by heart before tweaking the weights of your model.

In this article, we will walk through how traditional fine-tuning works and its shortcomings, what LoRA is, and some interesting properties of it.

✨This is a paid article. If you’re not a Medium member, you can read this for free in my newsletter: Qiubyte.

As the size of Large Language Models (LLM) increases to hundreds of billions, fine-tuning these beasts becomes a challenge. Traditionally, to fine-tune a model, we would need to update all of the model parameters. This is also known as Full Fine-Tuning (FFT). A close overview of how this method works can be seen in the diagram below.

An overview of how Full Fine-Tuning works. Note that this is for comprehensive purposes and in reality, FFT updates W in-place and replaces it with W’ (by Author).

There are a few trivial problems with this approach. First, the computational cost and resource requirements for FFT are substantial, as updating every parameter requires significant processing… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI