Understanding MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

Author(s): Hesam Sheikh

Originally published on Towards AI.

the math and intuition behind a novel parameter-efficient fine-tuning method
The outline of MoRA vs LoRA. (source: MoRA paper)

A recent, “MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning”, introduces a new method into the family of parameter-efficient fine-tuning (PEFT) and possibly a new alternative to the famous LoRA — Low-Rank Adaptation of Large Language Models.

In this article, we will walk through what problem MoRA is trying to solve, the basic idea behind it, and how it compares to LoRA.

✨This is a paid article. If you’re not a Medium member, you can read this for free in my newsletter: Qiubyte.

Note: this article assumes you have comprehensive knowledge about LoRA. If this is a new concept, I would suggest reading this easy-to-understand article about LoRA and its drawbacks.

We will go through LoRA (Low-Rank Adaptation of Large Language Models), and compare LoRA to Full Fine-Tuning.

pub.towardsai.net

Efficiently fine-tuning gigantic Large Language Models with hundreds of billions of parameters is an open area of research in machine learning. In the Full Fine-Tuning (FFT) method, we would need to update all the weights of a model in place.

An overview of how Full Fine-Tuning works. Note that this is for comprehensive purposes and in reality, FFT updates W in-place and replaces it with W’ (by Author).

This way of fine-tuning presents… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI