LLM Quantization: Quantize Model with GPTQ, AWQ, and Bitsandbytes

Author(s): Luv Bansal

Originally published on Towards AI.

This blog will be ultimate guide for Quantization of models, We’ll talk about various ways to quantizing models like GPTQ, AWQ and Bitsandbytes. We’ll discuss the pros and cons of each method (GPTQ vs AWQ vs Bitsandbytes), in the end, use quantized weights for efficient language model inference.

But before diving in, let’s begin by understanding what quantization is and why it’s necessary.

Quantization

A Large Language Model is represented by a bunch of weights and activations. These values are generally represented by the usual 32-bit floating point (float32) datatype.

The number of bits tells us something about how many values it can represent. Float32 can represent values between 1.18e-38 and 3.4e38, quite a number of values! The lower the number of bits, the fewer values it can represent.

As you might expect, if we choose a lower bit size, then the model becomes less accurate but it also needs to represent fewer values, thereby decreasing its size and memory requirements.

One of the most effective methods to reduce the model size in memory is quantization. You can see quantization as a compression technique for LLMs. In practice, the main goal of quantization is to lower the precision of the LLM’s weights, typically from 16-bit to 8-bit, 4-bit, or even 3-bit. However, we do not simply want to use a smaller bit variant but map a larger bit representation to a smaller bit without losing too much information.

Quantisation Techniques

There are three popular quantization methods for LLMs, we will discuss briefly about each of them, then we will look into how to use each of the techniques in code.

• GPTQ
• AWQ
• Bitsandbytes NF4

GPTQ Quantization

GPTQ is a post-training quantization (PTQ) method for quantization that focuses primarily on GPU inference and performance.

Post-training quantization (PTQ) is the sort of quantization, wherein the weights of an already skilled model are transformed to a lower precision without any retraining. This is a trustworthy and clean-to-put-into-effect method

It is based on one-shot weight quantization method based on approximate second-order information, that is both highly- accurate and highly-efficient.

The GPTQ algorithm makes it possible to quantize models up to 2 bits! However, this might come with severe quality degradation. The recommended number of bits is 4, which seems to be a great tradeoff for GPTQ at this time.

The idea behind the method is that it will try to compress all weights to a 4-bit quantization by minimizing the mean squared error to that weight.

Quantize Model to GPTQ

GTPQ requires a lot of GPU VRAM. For this experiment, we are using Nvidia A10 GPU which has 24 GB of memory and we are quantizing Llama 2–7B model and the VRAM consumption peaked at around 16 GB.

Quantization with GPTQ is also slow. It took 35 min with one A10, The quantization speed and VRAM/RAM consumption are the same for the 4-bit, 3-bit, and 2-bit precisions.

While you can’t quantize Llama 2 with GPTQ on the Google Colab free tier. Smaller models (<4B parameters) can be quantized with a colab-free tier.

I used the same snippet that TheBloke uses to quantify HF models to GPTQ. You can find the script here.

You can use the same TheBloke script easily with the below command.

Note: You need to install transformers optimum accelerate auto-gptq first

python ./quant_autogptq.py meta-llama/Llama-2-7b-chat-hf gptq_checkpoints c4 --bits 4 --group_size 128 --desc_act 1 --damp 0.1 --seqlen 4096

I’m simplifying the script above to make it easier for you to understand what’s in it.

To quantize with GPTQ, I installed the following libraries:

pip install transformers optimum accelerate auto-gptq

Importing libraries and loading dataset

from datasets import load_dataset
import os
import random
import torch
import numpy as np
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

model_id = "meta-llama/Llama-2-7b-hf"

tokenizer = AutoTokenizer.from_pretrained(model_id, 
 use_fast=True,)
def get_c4():
 
 traindata = load_dataset(
 'allenai/c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train', use_auth_token=False
 )
 seqlen = 4096
 trainloader = []
 for _ in range(self.num_samples):
 while True:
 i = random.randint(0, len(traindata) - 1)
 trainenc = tokenizer(traindata[i]['text'], return_tensors='pt')
 if trainenc.input_ids.shape[1] >= self.seqlen:
 break
 i = random.randint(0, trainenc.input_ids.shape[1] - self.seqlen - 1)
 j = i + seqlen
 inp = trainenc.input_ids[:, i:j]
 attention_mask = torch.ones_like(inp)
 trainloader.append({'input_ids':inp,'attention_mask': attention_mask})
 return trainloader

1. It specifies the pre-trained model ID (“meta-llama/Llama-2–7b-hf”) to load the tokenizer.
2. get_c4() function is to load the dataset which is used for quantization.

Note that TheBloke didn’t use the entire ‘c4’ dataset which is quite large and it won’t fit in 24GB A10 GPU (I tried it and was giving OOM or Out Of Memory Error while quantising). Instead, only a small part of the ‘c4’ dataset was used.

The quantization and serialization with Transformers

quantize_config = BaseQuantizeConfig(
 bits=bits,
 group_size=group_size,
 desc_act=desc_act,
 damp_percent=damp
)

torch_dtype = torch.float16

traindataset = get_c4()

model = AutoGPTQForCausalLM.from_pretrained(model_id, quantize_config=quantize_config,
 low_cpu_mem_usage=True, torch_dtype=torch_dtype)


model.quantize(traindataset, use_triton=self.use_triton, batch_size=self.batch_size, cache_examples_on_gpu=self.cache_examples)

3. It specifies the pre-trained model ID (“meta-llama/Llama-2–7b-hf”) that will be loaded and quantized.

4. It creates a BaseQuantizeConfig object, which configures the GPTQ quantization process:

• bits: Precision of the quantization. You can set 4, 3, and 2.
• group_size: The group size to use for quantization. The recommended value is 128 and -1 uses per-column quantization
• desc_act: Whether to quantize columns in order of decreasing activation size. Setting it to False can significantly speed up inference but the perplexity may become slightly worse. Also known as act-order.

Save quantized Model

output_dir = 'Llama-2-7b-hf-gptq'

model.save_quantized(output_dir, use_safetensors=True)
tokenizer.save_pretrained(output_dir)

5. Finally mention the path where you want to save quantize model weights, and from where quantized model will be loaded for inference.

Push Quantise Model to Hub

You can upload (push) the quantized language model to the Hugging Face Hub, Then, anyone can use it right away for their work. It’s also great for collaborative contributions in the open-source community.

from huggingface_hub import HfApi

api = HfApi()
api.upload_folder(
 folder_path="Llama-2-7b-hf-gptq",
 repo_id="luv2261/Llama-2-7b-hf-gptq",
 repo_type="model",
)

Check out the models I quantized using the method mentioned above here and feel free to use them for your own work.

Load and Inference quantized models from the U+1F917 Hub

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "luv2261/Llama-2-7b-hf-gptq"

model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

prompt = "<s>[INST] Write a tweet on future of AI [/INST]"
inputs = tokenizer(prompt, return_tensors="pt").to(0)

out = model.generate(**inputs, max_new_tokens=250, temperature = 0.6, top_p=0.95, tok_k=40)

print(tokenizer.decode(out[0], skip_special_tokens=True))

AWQ Quantization

AWQ (Activation-aware Weight Quantization) which is a quantization method similar to GPTQ. There are several differences between AWQ and GPTQ as methods but the most important one is that AWQ assumes that not all weights are equally important for an LLM’s performance.

In other words, a small fraction of weights will be skipped during quantization, which helps with the quantization loss.

As a result, their paper mentions a significant speed-up compared to GPTQ whilst keeping similar, and sometimes even better, performance.

Quantize Model to AWQ

AWQ performs zero point quantization down to a precision of 4-bit integers.

AutoAWQ integration with Transformers

Note:

• Some models like Falcon is only compatible with group size 64.
• To use Marlin, you must specify zero point as False and version as Marlin.

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = 'meta-llama/Llama-2-7b-hf'

quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
# Load model
model = AutoAWQForCausalLM.from_pretrained(
 model_path, **{"low_cpu_mem_usage": True, "use_cache": False}
)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
# Quantize
model.quantize(tokenizer, quant_config=quant_config)

Make compatible with Transformers and Save quantized Model

In order to make it compatible with transformers, we need to modify the config file and then save the quantized model.

from transformers import AwqConfig, AutoConfig
from huggingface_hub import HfApi

# modify the config file so that it is compatible with transformers integration
quantization_config = AwqConfig(
 bits=quant_config["w_bit"],
 group_size=quant_config["q_group_size"],
 zero_point=quant_config["zero_point"],
 version=quant_config["version"].lower(),
).to_dict()

# the pretrained transformers model is stored in the model attribute + we need to pass a dict
model.model.config.quantization_config = quantization_config
# a second solution would be to use Autoconfig and push to hub (what we do at llm-awq)

# Save quantized model
quant_path = 'Llama-2-7b-hf-awq'

model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
print(f'Model is quantized and saved at "{quant_path}"')

Uploading the quantized model to the hugging face hub and loading the model from the hub is pretty much like what I explained for GPTQ.

Bitsandbytes NF4

bitsandbytes is the easiest way for quantizing a model to 8 and 4-bit

It works in three steps:

1. Normalization: The weights of the model are normalized so that we expect the weights to fall within a certain range. This allows for more efficient representation of more common values.
2. Quantization: The weights are quantized to 4-bit. In NF4, the quantization levels are evenly spaced with respect to the normalized weights, thereby efficiently representing the original 32-bit weights.
3. Dequantization: Although the weights are stored in 4-bit, they are dequantized during computation which gives a performance boost during inference.

It represents the weights with 4-bit quantization but does the inference in 16-bit.

If you want to know more about how this quantisation works, I highly recommend to go though this blog

This is a wonderful technique but it seems rather wasteful to have to apply them every time we have to load the model.

Quantise model with Bitsandbytes

Using this quantisation is straightforward with HuggingFace. we just need to define a configuration for the quantization with Bitsandbytes:

from transformers import BitsAndBytesConfig
from torch import bfloat16

# Our 4-bit configuration to load the LLM with less GPU memory
bnb_config = BitsAndBytesConfig(
 load_in_4bit=True, # 4-bit quantization
 bnb_4bit_quant_type='nf4', # Normalized float 4
 bnb_4bit_use_double_quant=True, # Second quantization after the first
 bnb_4bit_compute_dtype=bfloat16 # Computation type
)

This configuration allows us to specify which quantization levels we are going for. Generally, we want to represent the weights with 4-bit quantization but do the inference in 16-bit.

Loading the model in a pipeline is then straightforward:

from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline

# Zephyr with BitsAndBytes Configuration
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-alpha")

model = AutoModelForCausalLM.from_pretrained(
 "HuggingFaceH4/zephyr-7b-alpha",
 quantization_config=bnb_config,
 device_map='auto',
)

# Create a pipeline
pipe = pipeline(model=model, tokenizer=tokenizer, task='text-generation')

prompt = '''
<U+007CsystemU+007C> \
You are a friendly chatbot who always responds in the style of a pirate.\
</s>\
<U+007CuserU+007C>\
Write a tweet on future of AI. </s>\
<U+007CassistantU+007C>
'''

outputs = pipe(
 prompt, 
 max_new_tokens=256, 
 do_sample=True, 
 temperature=0.7, 
 top_p=0.95
)
print(outputs[0]["generated_text"])

Bitsandbytes vs GPTQ vs AWQ

A quick camparition between Bitsandbytes, GPTQ and AWQ quantization, so you can choose which methods to use according to your use case.

Bitandbytes

• This method quantise the model using HF weights, so very easy to implement
• Slower than other quantisation methods as well as 16-bit LLM model.
• This is a wonderful technique but it seems rather wasteful to have to apply them every time we have to load the model.
• Take longer time to load the models weights

GPTQ

• Much Faster as compared to Bitandbytes
• New model architectures are promptly supported in AutoGPTQ

Challenges

• Need to quantise the model weights to GPTQ weights beforehand to use it in production.
• High computation to quantise the model
• Around ~ 16GB GPU memory required to quantise 7B parameter model

AWQ

• Paper mentions a significant speed-up compared to GPTQ whilst keeping similar, and sometimes even better, performance

Limitations

• Newer architectures like Gemma or DeciLM doesn’t support in AWQ quantization yet.

Final thoughts

This article discusses various techniques to quantize models like GPTQ, AWQ and Bitsandbytes. It looks at the pros and cons of each method (GPTQ vs AWQ vs bitsandbytes), explains quantizing hugging-face model weights using these methods and finally use quantize weights for LLM inference. These techniques can help you create and use Large Language Models more effectively in real-world applications.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI