Alibaba Open Sources EMO: Make Your Avatar Videos Sing Naturally

Author(s): Meng Li

Originally published on Towards AI.

https://humanaigc.github.io/emote-portrait-alive/

For us, generating a single 2D image with technologies like Stable Diffusion or Midjourney is no longer a challenge.

However, the complexity arises when we attempt to stitch these consecutively generated 2D images together to create a smooth video.

In the realm of 3D generation technology, avatar technology has always been a focal point.

This technology can be broadly divided into two main directions: creation and animation of digital avatars.

Today, we’re delving into a new advancement in the animation of digital avatars.

Recently, Alibaba has open-sourced a project named EMO: Emote Portrait Alive.

Traditionally, avatar videos generated appear stiff and unnatural, resembling a simple compilation of static facial expressions.

However, the advent of EMO technology has changed this narrative.

EMO has successfully produced singing avatar videos, achieving a natural and lively effect.

So, how does EMO technology accomplish this?

And what is the quality of the avatar videos it generates?

Let’s explore together.

https://humanaigc.github.io/emote-portrait-alive/

To create lifelike talking head videos, the EMO framework takes a unique approach by adopting a direct audio-to-video synthesis strategy.

Unlike traditional, cumbersome processes that rely on 3D models or facial landmarks, EMO’s simplicity raises the question: can such simplicity truly achieve the desired outcome?

At the core of the EMO framework is a text-to-image model known as Stable Diffusion… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI