Exploration of Parameters-efficient fine-tuning methods (LoRA/MoRA/DoRA) in LLM

Author(s): Anish Dubey

Originally published on Towards AI.

Introduction

Pre-trained models using extensive general domain datasets have demonstrated impressive generalization abilities, benefiting a wide range of applications, from natural language processing (NLP) to multi-modal tasks. Adapting these general models for specific downstream tasks typically involves full fine-tuning (FT), which retrains all model parameters. However, as models and datasets grow in size, the cost of fine-tuning the entire model becomes very high.

To address this issue, parameter-efficient fine-tuning (PEFT) methods have been introduced, enabling fine-tuning with only a small subset of parameters. Among these, LoRA/DoRA/MoRA have been introduced to solve the above problem.

The below post aims to help distinguish various techniques like (LoRA/DoRA/MoRA) and give a high-level overview and a general intuition on how it works.

What is LoRA?

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning (PEFT) method designed to adapt pre-trained models for specific tasks without the need to retrain all the parameters. This approach aims to mitigate the high computational cost and resource demands associated with full fine-tuning, especially as model sizes grow.

How does LoRA work?

• Low-Rank Matrices: LoRA assumes that the weight updates during fine-tuning can be well-approximated by low-rank matrices. Instead of updating the full-weight matrix W, LoRA decomposes the update into two smaller matrices A and B. For example, if the original weight layer has 4096 × 4096 = 16,777,216 parameters, LoRA with r=8 has 4096×8 + 8×4096 = 65,536 parameters.
• Efficiency: By updating only these smaller matrices, LoRA reduces the number of trainable parameters significantly. This makes the fine-tuning process more efficient in terms of both computation and memory.

An example demonstrating how LoRA works is shown in the diagram below:

• Key, Query, and Value weight matrices are broken into B (5X1) and A (1X3) matrices, respectively. Here, rank = 1.
• Key = Key + B1*A1, here the Key matrix is kept frozen during backward gradient update while B1 and A1 are only updated.

What is the concern with the approach below?

One of the main concerns is that matrices are projected into smaller dimensions and, hence, lose some information along the way.

What is DoRA doing?

Weight-Decomposed Low-Rank Adaptation (DoRA) decomposes the pre-trained weight into two components: magnitude and direction. For fine-tuning, it specifically employs LoRA for directional updates, efficiently minimizing the number of trainable parameters.

Mathematical point of view: Each column can be represented by a vector. Ref image below. If each column of weight matrices can be represented by magnitude and direction:

The authors found that while doing full fine-tuning, gradient updates were only changing the direction of the column vector while keeping the magnitude almost constant. While LoRA was updating both magnitudes as well as direction components.

Breaking down column vectors into 2 components gives flexibility in terms of updating only the directional component, which resembles more closely to full fine tuning.

What is MoRA doing?

The concept of MoRA is similar to LoRA, but instead of breaking the weight matrix into smaller dimensions, it is broken down into small square matrices.

For example, if the original weight layer has 4096 × 4096 ~= 16M parameters, LoRA with r=8 has 4096×8 + 8×4096 = 65,536 parameters. With MoRA, we can reduce the dimension to r=256, i.e. 256 × 256 = 65,536. In both cases, parameters are the same but authors claim to have higher learning representation as compared to LoRA.

According to the paper, one of the limitations of LoRA is the inability to memorize huge amounts of data:

“One plausible explanation for this limitation observed with LoRA could be its reliance on low-rank updates. The low-rank update matrix, ∆W , struggles to estimate the full-rank updates in FFT, particularly in memory-intensive tasks like continual pre-training that require memorizing domain-specific knowledge.”

How to validate this hypothesis?

The author took a random UUID (randomly generated unique identifier) and fed it into a pre-trained model during fine-tuning. When compared with FFT (full fine-tuning), LoRA was not able to memorize the data, while MoRA did comparatively better.

Refer image below, which shows the comparison of how MoRA was able to learn better than LoRA

Adding the illustration explaining it below:

Final conclusion

All the above variant somewhere explores reducing the dimensionality of the weight matrix either by projecting to smaller dimensions, square dimensions or bisecting them into magnitude and direction and applying LoRA on direction component. The trade-off is on the amount of information lost while retaining the maximum information.

One of the Future Exploration

Since a lot of iterations around DoRA/MoRA/LoRA is coming, each one has its own positives/negatives. One exploration is if each layer of the transformer (attention + multilayer perceptron) can experiment with different techniques.

Example:

Outer layers can experiment with MoRA where architecture learns basic info and sufficient memoization is required.

Inner layers can experiment with DoRA where only nuanced information is being learned by the network, and hence adjusting only directional components may be sufficient.

Refer diagram below for illustration:

References:

• DoRA: https://arxiv.org/abs/2402.09353
• MoRA: https://arxiv.org/pdf/2405.12130v1
• LoRA: https://arxiv.org/abs/2106.09685

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI