The Generative AI Model Map

Author(s): Ayo Akinkugbe

Originally published on Towards AI.

Introduction

With the commercialization of the GPT model in 2022, generative AI (artificial intelligence) became popular. However large language models — the category of generative models GPT belongs to are only a subset of a family of generative models. The goal of this article is to delineate the various types of generative models and map out their differences and use cases.

A generative model, a type of AI models, is an artificial intelligent system that learns from examples of data (for example — like lots of pictures or sentences) and then creates new, similar examples. It’s like showing a robot thousands of cat photos, and then asking it to draw a new, unique cat photo by itself. Generative models can be demarcated into two categories — Explicit Density Models and Implicit Density Models.

Explicit Density Model

Explicit density models are models that explicitly represents and calculates a probability distribution p(x) for data. In the case of generative models, they know exactly how likely or probable each piece of data generated is.

For instance, imagine you have a recipe book where, for any ingredient combination, you know exactly how often that recipe created will be picked. You not only make new recipes (i.e generate data) but also know how likely each one is ( i.e model density). The intuition behind Explicit density generative models isn’t just about generating things, but also knowing how probable each thing is according to what has been learned from the data.

The goal of such model is to model or approximate the probability distribution Pmodel​(x) of data x. Generally these type of models are trained by maximizing the likelihood of the data

where:

• θ are model parameters
• xi​ is a data point
• Pmodel​(x; θ) is a density function (which can actually be computed.)

Explicit Density Generative models can be further broken into Tractable Density and Approximate Density

Tractable Density Models

Tractable density models are models where the math is easy enough (fast and exact) for calculating the probability exactly for any generated sample. For example, given some data, you can plug it into a formula and instantly get its probability under the model with no tricks or shortcuts. The calculations are efficient, and there’s no approximation or guessing involved.

Popular examples of tractable density models include:

• Autoregressive Models (like GPT, PixelCNN): These models break complicated predictions into smaller, manageable steps. For a sentence, they compute the chance of each word, given the ones before, and multiply them together.
• Normalizing Flow Models: These models transform data in a reversible way, to compute the new probability exactly (using the change-of-variables formula from math).
• Naive Bayes or Gaussian Mixture Models: These models offer a simple formula for computing the probability.

Approximate Density

These models find it hard or almost impossible to calculate the exact likelihood of generated samples and try to estimate instead. Instead of offering an exact answer, these models offer a close-enough answer using clever techniques, such as sampling, variational inference, or optimization. It’s cheating one’s way into a useful probability estimate, but it might not be 100% accurate.

Popular examples of approximate density models include:

• Variational Autoencoders (VAEs): They can’t compute the real probability for every sample exactly, so they use an “evidence lower bound” (ELBO) which is a safe, close guess that’s good enough for training and comparing samples.
• Energy-Based Models: Often can’t normalize their probabilities exactly, so they use sampling or approximation tricks.
• Diffusion Models: Diffusion models begin data generation from noise and de-noise. Even though traditional diffusion models are implicit, many new variants of Diffusion models (like Denoising Diffusion Probabilistic Models (DDPM) ) approximate the true likelihood because tracking how noise changes over many steps is too complex for exact calculations.

Implicit Density Model

Implicit density models are models that are able to generate realistic samples from the data distribution without calculating the exact probability of each sample. These models are great at making realistic things, but they don’t know the exact “likelihood” of anything they create. They can make amazing new images or text, but if you ask “how likely is this photo?” they just shrug — they don’t have that number. Implicit density generative models do not provide a numerical measure of how likely or realistic a specific sample is, according to their learned data distribution.

It’s like a very creative chef who keeps trying new dishes and learning only from taster feedback (“Yum!” or “Yuck!”), but never writes down or even calculates how often any one recipe is made. The chef just wants the dishes to taste real. The intuition behind Implicit density generative models focuses on generating realistic samples but not on measuring probability for each creation.

The goal here is to generate realistic samples from the data distribution, without calculating the exact probability of each point. Usually, these models are trained so that the generated data x “fooled” some test or metric into thinking it is real. The goal for training such a model (GANs — Generative Adversarial Network in this case) looks like

Implicit Density Generative models can be further broken into Generative Adversarial Networks and Score Based Generative Models

Implicit density models generate realistic samples from the data distribution without calculating the exact likelihood of each sample.

Generative Adversarial Networks (GANs)

GANs are an interesting type of generative models with lots of pros and cons and a section in this post would not do them justice. However the basic intuition is two networks “compete”: a generator creates fake data and a discriminator tries to tell real from fake. The generator never explicitly computes the probability of a sample — it just learns to make samples that fool the discriminator. There are many variants of GANs each addressing various problems these types of models have. These variants include:

• Conditional GANs — Unlike vanilla GANs which just generate random samples, conditional GANs allows for control what the GAN generates by feeding in a label or some extra data — the “condition”).Both the generator and discriminator get extra input: a label, a class, or some side information. They solve the problem of targeted generation, allowing for generation images of a specific class. For example “Generate a picture of a dog” vs. “Generate a picture of a cat.”
• CycleGAN — CycleGAN learns to convert images from domain A to domain B without needing paired examples. For example: horses to zebras (viceversa), summer to winter (viceversa). It uses two generators (A→B and B→A) and cycle consistency loss — if you translate A→B then B→A, you should get the original image back. It solves the problem of unsupervised domain transfer which is useful when direct paired data is unavailable ( for example — photos of the same horse as both a horse and a zebra). It is often used in style transfer (photo to painting, summer to winter) and data augmentation for domains where paired samples are expensive or impossible.
• StyleGAN — StyleGAN introduces a style-based generator architecture, letting users finely control high-level and low-level details (for example pose vs. freckles in faces). It injects “style” at different layers in the generator, allowing smooth manipulation of image attributes allowing for more photorealistic image creation
• BigGAN — BigGAN is designed for large-scale, high-quality image generation by using much larger networks and datasets. It leverages more filters, deeper models, and tricks for stable training at scale. It solves the problem of generated sample quality and diversity at scale and pushes boundaries of how large/diverse a GAN-generated image can be while keeping realism.

In generative adversarial networks, two networks compete — a generator creates fake data and a discriminator tries to tell real from fake.

Score-Based Generative Models

Score-Based Generative Models start from random noise and repeatedly denoise, guided by a learned process (score function). The score function acts as a compass through out the generative process towards a clear, realistic image/data sample. Mathematically, the score is the gradient of the log-probability density — it points toward higher-probability regions of the data. It is important to call out that Score-based models know which direction to go for higher probability, but not the actual probability at any point — hence they’re implicit density models.

Often, sampling with score-based models uses Langevin Dynamics. You take a gradient ascent step plus a bit of randomness (to help explore the space and avoid getting stuck). The score function is the gradient of the log-probability ∇x ​log p(x). Following the gradient increases the likelihood that the generated data x is a real example.

Where ϵ is the step size, and ξt​ is random noise.

It is important to call out that Score-based models know which direction to go for higher probability, but not the actual probability at any point — hence they’re implicit density models.

Conclusion

Explicit models such as normalizing flows, autoregressive models, and VAEs are preferred when it’s important to measure the probability or likelihood of data. In contrast, implicit models like GANs, while capable of producing highly realistic images, face challenges like training instability and mode collapse, limiting their broader adoption. ChatGPT exemplifies the success of explicit models. As an autoregressive transformer, it predicts each word based on a well-defined probability distribution over possible next words, allowing it to assign a likelihood to entire sentences.

As transformer-based models continue to evolve and gain popularity, exploring the diverse architectures — such as encoder-only, decoder-only, and encoder–decoder (seq2seq) transformers — reveals even more possibilities for exploring both generative AI options.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI