Classifying the Unstructured: A Guide to Text Classification with Representation and Generative Models

Author(s): Shivam Dattatray Shinde

Originally published on Towards AI.

This article will delve into the various methodologies to perform text classification using transformer-based models, explaining their principles, applications. We’ll explore both representation-focused and generative approaches, leveraging the flexibility and power of transformer architectures to tackle unstructured text data.

Agenda

1. What are representation language models?
2. What are generative language models?
3. Text Classification Methods
4. Text classification using representation language models
5. Text classification using generative language models

What are Representation Language Models?

The original transformer architecture was designed as an encoder-decoder model primarily for machine translation tasks. However, it was not well-suited for other tasks like text classification.

To address this limitation, a new architecture called Bidirectional Encoder Representation of Transformer (BERT) was introduced. BERT focuses on text representation and is derived from the encoder component of the original transformer. Unlike the original transformer, BERT does not include a decoder.

BERT is specifically designed to create contextualized embeddings, which outperform traditional embeddings generated by models like Word2Vec. Contextualized embeddings take into account the context in which words appear, resulting in more meaningful and versatile representations of text.

How is BERT Trained?

BERT uses a masked language modeling technique during training. This involves masking certain words in a sentence and training the model to predict the masked words based on the surrounding context.

For example, consider the input:
“The lake is ____.”
The model is trained to predict words such as “beautiful,” “serene,” or “cool” based on the context provided by the rest of the sentence.

What are Generative Language Models

Decoder-only architectures, like the encoder-only BERT architecture, are highly effective in specific applications. One of the most notable examples of a decoder-only architecture is the Generative Pretrained Transformer (GPT).

Generative language models operate by taking text as input and predicting the next word in the sequence. While their primary training objective is to predict the next word, this functionality alone is not particularly useful in isolation. However, these models become significantly more powerful when adapted for tasks such as serving as a chatbot.

Here’s how a chatbot built on a generative language model functions:
When a user provides input text, the generative language model predicts the next word in the sequence. This predicted word is appended to the user’s original input, forming a new, extended text sequence. The model then uses this updated sequence to predict the next word. This process repeats iteratively, generating responses word by word.

Text-Classification Methods

Text classification using representation language models

Using Task-Specific Models
A task-specific model, like BERT, is trained directly for a specific task, such as text classification.

Using Embedding Models

Using Classification Model
This approach involves converting input text tokens into contextual embeddings using representation models like BERT. These embeddings are then fed into a classification model.

This process has two steps: the BERT model generates embeddings, while only the classification model is trainable. BERT itself remains fixed during training.

Using Cosine Similarity

This method entails generating embeddings for both the input text to be classified and the classification labels. Next, the cosine similarity between the input text embedding and each label embedding is calculated. The input text is then assigned to the label with the highest similarity score.

Text classification using generative language models

Text classification using generative language models differs significantly from that of representational language models. Generative models are sequence-to-sequence models, producing output in the form of text or sentences rather than directly assigning labels.

For example:
If the input text is “Best movie ever!”, a generative language model might predict “The sentiment of the movie is positive.” However, unlike representational models, generative models don’t automatically provide labels without explicit instructions.

If you simply input “Best movie ever!” into a generative model, it won’t inherently understand what to do. To classify the sentiment of the input, you need to provide a clear instruction, such as “Classify the input movie sentiment as Positive or Negative.”

Moreover, the model’s classification accuracy heavily depends on the clarity of the instruction. Ambiguous or unclear instructions can lead to incorrect or irrelevant outputs.

Explore how varying prompts lead to different classification outputs from the generative language model in the diagram below.

Outro

Thank you so much for reading. If you liked this article, don’t forget to press that clap icon. Follow me on Medium and LinkedIn for more such articles.

Are you struggling to choose what to read next? Don’t worry, I have got you covered.

and more …

Have a great day!

References

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI