Video Reconstruction using Transformers

Author(s): Sarvesh Khetan

Originally published on Towards AI.

Here we can use any type of generative models like Autoencoders / Variational Autoencoders / GANs / …. Diffusion Models / … below I have shown how to reconstruct videos using Autoencoders and Diffusion Models.

Before going through this article I would recommend going through generative models for images and transformers for videos articles since this article will use a lot of concepts from these !!

Autoencoders (using Transformers) for Video Reconstruction

The below architecture is very intuitive if you have understood following two architectures

• Autoencoders Transformers for Image Generation
• Transformers for Video Classification

As discussed in Transformers for Video Classification blog, above architecture is highly compute intensive since it calculates attention between all the patches and hence instead we can use spatial and temporal attentions to reduce computations. Architecture diagram below

Diffusion Model (using Transformers) for Video Reconstruction — Diffusion Transformers (DiT)

As we know that in DDPM we use autoencoders and hence we will use almost the same architecture as seen above, just that we will add time information to the model. As seen here, there are several ways to add time information to the model, but cross attention works best and hence in below architecture I have shown same

As discussed in Transformers for Video Classification blog, above architecture is highly compute intensive since it calculates attention between all the patches and hence instead we can use spatial and temporal attentions to reduce computations. Architecture diagram below

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI