Extending Context Length of an LLM: Intuition, Implementation & Interview Questions

Author(s): Harsh Maheshwari

Originally published on Towards AI.

Understand multiple methods like Positional Interpolation, NTK awareness, and Dynamic NTK for extending context length.
Photo by Edward Howell on Unsplash

To begin with, let’s understand the meaning of context length of a Large Language Model (LLM). A lot of people assume that the context length is equal to the maximum number of input tokens that an LLM can handle, but that is not true. The context length of an LLM is the maximum number of tokens an LLM can handle at a time. For example, if the context length of an LLM is 1024 and the length of a given input sequence is 200, then the LLM can produce a maximum of 824 tokens (1024–200) for that input sequence.

Before delving into the technicalities of extending the context length of LLMs, it is important to first understand the motivation behind the same. Many open-source models have a limited context length, such as 2048 tokens for Llama, 4096 tokens for Llama 2, 8192 tokens for Llama 3, and 8192 tokens for Mixtral. This poses a significant challenge for tasks like summarization and retrieval-augmented generation. These tasks require the model to process and understand large amounts of text in a single pass. The current context length restriction acts as a bottleneck, hindering the models’ ability to perform these… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI