Natural Language Generation (X2Text) Tasks

Author(s): Sarvesh Khetan

Originally published on Towards AI.

Table of Contents

1. Image2Text
2. Speech2Text
3. Video2Text

Image2Text i.e. Image Captioning

Idea

Researchers came up with a very innovative idea to solve this problem. They framed this problem as a next word prediction task (with only input word as <START>) conditioned on image information. Obviously instead of next word prediction this could be next character prediction / next subword prediction but for simplicity we will consider next word prediction problem.

Now we know that for next word prediction we can use any sequence model like (you cannot use bidirectional models like transformer encoder)

• RNN (unidirectional) or stacked
• LSTM (unidirectional) or stacked
• GRU (unidirectional) or stacked
• Decoder Transformer (unidirectional) or stacked

Below I have shown how to add image information to LSTM and Transformers like architectures!!

Image Conditioned LSTM for Next Word Prediction

Method 1

Here idea is to condition the LSTM by adding the image information to the first hidden state of the LSTM as shown below.

Method 2

Instead of feed the image embedding only to the first hidden state of LSTM you can send it to each and every hidden state of the LSTM at the every timestamp but it was found that this performs worse compared to above approach.

Method 3

Here idea to add image information is by using image token as first input to the LSTM network as shown in the below diagram.

multimodal_ai/image_captioning.ipynb at main · khetansarvesh/multimodal_ai

Contribute to khetansarvesh/multimodal_ai development by creating an account on GitHub.

github.com

Image Conditioned Transformers for Next Word Prediction

Above I have shown conditioning the transformer using images via self attention instead we could have also use cross attention to condition the image information.

Applications — OCR

We already saw one method to solve OCR here. Following is another idea which worked much better compared to this :

• Use object detection model to identify bounding boxes where text is present in the image
• Now create an image captioning model which can take input such images and output text inside it.

Audio2Text / Speech2Text eg audio transcription (also called Speech Recognition or Automatic Speech Recognition (ASR) System)

The idea remains same as we saw in Image2Text i.e. we will treat this as a next word prediction task (with only input word as <START>) conditioned on audio information.

Whisper by OpenAI was built on above idea where they used a transformer based model for next word prediction which was conditioned (using cross attention) by speech embeddings. They got speech embeddings by converting speech into images and then using ViT to convert these images into embeddings !!

Video2Text i.e. Video Captioning

The idea remains same as we saw in Image2Text i.e. we will treat this as a next word prediction task (with only input word as <START>) conditioned on video information.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI