Author(s): Lorenzo Zarantonello

Originally published on Towards AI.

The increasing costs and diminishing returns of training LLMs create the ground to optimize small AI models.

This member-only story is on us. Upgrade to access all of Medium.

Until recently, big tech and researchers tried to build larger, more complex AI models.

Photo by imgix on Unsplash

The underlying assumption was that bigger is inherently better.

However, a new approach is appearing in the model training landscape. It seems smaller, more focused models can compete and outperform bigger models at specific tasks.

For years, most of the AI community focused on the size of models, especially adding more parameters and larger datasets to improve performance.

However, frontier AI models are showing a few clear trends :

Converging PerformanceIncreasing Training Costs For Marginal Improvements

Furthermore, big LLMs cannot be used on users’ hardware simply because they are too resource-intensive.

Frontier Models can be optimized through Low-Rank Adaptation (LoRA) and quantization, among others.

Low-rank adaptation reduces the computational resources needed for fine-tuning.Quantization compresses models to decrease the model size without losing too much accuracy.

GPT-4o mini is a good example of a model that is smaller and cheaper than its parent but not more performant.

GPT-4o mini is “an order of magnitude more affordable than previous frontier models and more than 60% cheaper than GPT-3.5 Turbo.”

GPT-4o mini introduction

We start to see a few examples of smaller and better AI models.

A… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI