From Words to Vectors: Exploring Text Embeddings

Author(s): Shivam Dattatray Shinde

Originally published on Towards AI.

This article will guide you through the various techniques for transforming text into formats that machines can understand. We’ll cover everything from basic encoding methods to more complex approaches like embeddings.

Agenda

1. What are large language models?
2. Why is it important to convert words into numbers or numerical arrays
3. What are the various methods for converting words into numbers or numerical arrays? : Bag-of-Words, TF-IDF, Hashing, Embeddings, Contextualized Embeddings

What are Large Language Models (LLMs)?

In simple terms, large language models (LLMs) are advanced deep learning models designed to handle various language-related tasks with high accuracy.

These models are typically trained on massive datasets using transformer-based architectures. LLMs can perform a wide range of tasks, including Language translation, Text summarization, Text classification, Named entity recognition, Answering questions, etc.

Why Convert Words to Numbers or Arrays of Numbers?

Computers are inherently designed to process numbers, not text. Therefore, the first step in training any large language model (LLM) is to convert text into numerical representations or arrays of numbers.

There are various methods to achieve this, each with its strengths and weaknesses. The most popular techniques include:

1. Simple Bag-of-Words
2. Term Frequency — Inverse Document Frequency (TF-IDF)
3. Hashing
4. Embeddings
5. Contextualized Embeddings

Let’s dive into each method to understand how they work.

Simple Bag-of-Words

Bag-of-Words is a straightforward text encoding method. It works by counting how many times each word appears in a given document.

Let’s break it down with an example:

Training Data:
"I just saw a cat and a dog together"

The first step is to create a vocabulary from the training data:
Vocabulary: ['I', 'just', 'saw', 'a', 'cat', 'and', 'dog', 'together']

Next, we encode a new sentence using this vocabulary:
New Sentence: "I saw a cat and saw a dog"

In this sentence:

• 'I' appears 1 time
• 'saw' appears 2 times
• 'a' appears 2 times
• 'cat' appears 1 time
• 'and' appears 1 time
• 'dog' appears 1 time
• Words like 'just' and 'together' do not appear.

We map these frequencies to their corresponding positions in the vocabulary array. For words not present in the sentence, we assign a value of 0.

Encoded Representation:
[1, 0, 2, 2, 1, 1, 1, 0]

Bag-of-Words is a simple and easy-to-understand encoding method. However, it has some limitations: it does not take into account the order of words or the semantic meaning of the words in the document.

TF-IDF: Term Frequency — Inverse Document Frequency

TF-IDF is an improvement over simple word count (as used in Bag-of-Words) for encoding text. While word count provides a starting point, it often gives too much weight to common words like “a,” “an,” and “the,” which don’t contribute much to understanding the meaning of the text.

To address this, TF-IDF assigns scores based on the importance of words in a document relative to their appearance across all documents. It is calculated as:

TF-IDF = Term Frequency (TF) × Inverse Document Frequency (IDF)

• Term Frequency (TF): Counts how many times a word appears in a specific document.
• Inverse Document Frequency (IDF): Measures how common or rare a word is across all documents.

This approach assigns higher scores to words that are frequent within a single document but rare across the overall collection of documents. As a result, it prioritizes important words (e.g., keywords) and reduces the weight of common stopwords like “a,” “an,” and “the.”

Hashing

The Bag-of-Words and TF-IDF approaches can be effective, but they often result in a very large vocabulary, which significantly increases memory requirements for these encodings.

To address this, the Hashing method can be used. In this approach, words are hashed (using a one-way function) into integers. This eliminates the need for a predefined vocabulary and allows us to use an encoding vector of any desired length.

However, this method has a limitation: since hashing is one-way, it is impossible to retrieve the original words from the encoded vector. While this may be a drawback in some scenarios, it is not an issue for supervised tasks like text classification.

Tokens and Embeddings

Understanding Tokens

Before learning about embeddings, it’s important to understand tokens.

In natural language processing (NLP) tasks, tokens are the fundamental units of text. These can be categorized into four types frequently used in NLP:

1. Word tokens (e.g., ‘I’, ‘am’, ‘batman’)
2. subword tokens, (e.g., ‘I’, ‘am’, ‘bat’, ‘man’)
3. character tokens, (e.g., ‘I’, ‘a’, ‘m’, ‘b’, ‘a’, ‘t’, ‘m’, ‘a’, ‘n’)
4. byte tokens.

Understanding Embeddings

Now, let’s understand the word embeddings.

“A word embedding is a learned representation for text where words that have the same meaning have a similar representation.[3]”

For instance,

Words representing two fruits, such as “orange” and “apple”, will have similar embeddings.

In contrast, the embedding for a fruit like “orange” will differ significantly from a word like “chair” (representing furniture).

Word embeddings also allow for interesting operations between words. For example:

king — man + women = queen

Embeddings can be created at various levels:

1. Word-Level Embeddings: Created using word tokens.
2. Subword-Level Embeddings: Created using subword tokens.
3. Character-Level Embeddings: Created using character tokens.
4. Byte-Level Embeddings: Created using byte tokens.

Methods to Create Embeddings

There are two ways to create these embeddings:

1. Training Embeddings Alongside a Neural Network: This method involves learning embeddings during the training process using an embedding layer
2. Training Embeddings Separately: This approach trains embeddings independently of the neural network. Two well-known methods in this category are: word2vec and Global Vectors for Word Representation or GloVe

word2vec

word2vec is a method for learning word embeddings from text corpus. Two different learning models were introduced as a part of word2vec approach to learn word embeddings.

1. Continuous Bag of Words (CBOW) Model
2. Continuous Skip-gram Model

CBOW model learns the embeddings by predicting the current word based on its context (aka its neighbors). On the other hand, continuous skip-gram model predicts the context words given the current word.

Both of these models learn the word embeddings using the context. This context is decided by the window of neighboring words. The size of context is a hyperparameter.

GloVe

GloVe algorithm creates the embeddings using the matrix factorization.

Using Embeddings

When working on NLP projects, you have several options for incorporating embeddings. Let’s explore these options in detail:

1. Learning the embeddings

This approach involves training embeddings using algorithms like Word2Vec. Typically, this requires a very large text corpus (millions or even billions of words) to train effectively. There are two main methods for learning embeddings:

a. Learning embeddings independently: In this method, a model is trained specifically to learn embeddings, which are then saved for future use. This approach is ideal when you plan to reuse the embeddings multiple times. The embeddings learned in this manner are general-purpose and can be applied to various tasks.

b. Learning embeddings jointly: Here, embeddings are learned alongside a task-specific model. These embeddings are tailored to the specific task and cannot be reused for other purposes (Can be used for other tasks, but probably won’t give good performance for the task). Unlike general-purpose embeddings, these are task-specific and optimized for the problem at hand.

1. Reusing the embeddings

If you lack the resources or time to train embeddings from scratch, you can use pre-trained embeddings. Many researchers make these embeddings freely available for public use. You can incorporate them into your project in one of two ways:
a. Freezed Embeddings: In this method, the embeddings remain static during training and are not updated. This approach is suitable when the pre-trained, general-purpose embeddings are a good fit for your use case and yield satisfactory results.

b. Updated Embeddings: In this approach, the embeddings are fine-tuned during training along with the task-specific model. Updating the embeddings often leads to better performance, as they become tailored to your specific task or dataset.

Contextualized Embeddings

Static vs. Contextualized Embeddings

Embeddings created using algorithms like Word2Vec or GloVe are static, meaning that the embedding for a particular word remains the same regardless of its context. While this can be useful in some cases, it can also lead to limitations.

For example,

Consider the following two sentences:

1. River bank is very far from here.
2. Bank opens at 10 in the morning.

In these sentences, the word “bank” has different meanings — one refers to the edge of a river, while the other refers to a financial institution. However, static embeddings created by models like Word2Vec or GloVe assign the same embedding to “bank” in both contexts.

To address this limitation, contextualized embeddings are used. Unlike static embeddings, contextualized embeddings consider the context in which a word appears and adjust the embedding accordingly.

Understanding Contextualized Embeddings

To create contextualized embeddings, attention mechanisms are used.

Attention helps determine how strongly each word in the sentence is related to the current word. This relationship is incorporated into the word’s embedding, allowing it to adapt to the specific context in which it is used.

As a result, the embedding of the word “bank” in the first sentence (referring to a riverbank) will differ from its embedding in the second sentence (referring to a financial institution).

Creating Contextualized Embeddings

Contextualized embeddings are generated using transformer-based models like BERT (Bidirectional Encoder Representations from Transformers). These models leverage attention mechanisms to capture the nuanced meaning of words based on their context, making them more effective for complex NLP tasks.

Outro

Thank you so much for reading. If you liked this article, don’t forget to press that clap icon. Follow me on Medium and LinkedIn for more such articles.

Are you struggling to choose what to read next? Don’t worry, I have got you covered.

Beyond Labels: The Magic of Autoencoders in Unsupervised Learning

In a world where labeled data is often scarce, autoencoders provide a powerful solution for extracting insights from…

pub.towardsai.net

and one more..

Efficient Backpropagation: Exploring the Role of Jacobian Matrices and the Chain Rule

In this article, I will tough upon the topic of Jacobian matrix and how it is used in the back-propagation operation of…

ai.plainenglish.io

References:

[1] https://learning.oreilly.com/library/view/hands-on-large-language/9781098150952/

[2] https://machinelearningmastery.com/prepare-text-data-machine-learning-scikit-learn/

[3] https://machinelearningmastery.com/what-are-word-embeddings/

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI