Extracting Features from Text Data
Last Updated on January 6, 2023 by Editorial Team
Author(s): Bala Priya C
Natural Language Processing
Part 3 of the 6 part technical series onย NLP

Hey everyone! ๐ This is part 3 of the 6 part NLPย series;
Part 1 of the NLP series introduced basic concepts in Natural Language Processing, ideallyย NLP101;
Part 2 covered certain linguistic aspects, challenges in preserving semantics, understanding shallow parsing, Named Entity Recognition (NER) and introduction to languageย models.
In this part, we seek to cover the Bag-of-Words Model and TF-IDF vectorization of text, simple feature extraction techniques that yield numeric representation of theย data.๐ฐ
Understanding Bag-of-Words Model
A Bag-of-Words model (BoW), is a simple way of extracting features from text, representing documents in a corpus in numeric form asย vectors.
A bag-of-words is a vector representation of text that describes the occurrence of words in a document.
Why is it called a โbagโย ?๐ค
It is called a โbagโ of words, because any information about the order or contextual occurrence of words in the document is discarded.

The BoW model only takes into account whether a word occurs in the document, not where in the document. Therefore, itโs analogous to collecting all the words in all the documents across the corpus in a bagย ๐
The Bag-of-Words model requires the following:
- A vocabulary of known words present in theย corpus
- A measure of the presence of known words, either number of occurrences/ frequency of occurrence in the entireย corpus.
Each text document is represented as a numeric vector, which each dimension denoting a specific word from the corpus. Letโs take a simple example as shownย below.
# This is our corpus
It was the best of times,
it was the worst of times,
it was the age of wisdom,
it was the age of foolishness,
Step 1: Collect theย data
We have our small corpus, the first few lines from โA Tale of Two Citiesโ by Charles Dickens. Letโs consider each sentence as a document.
Step 2: Construct the vocabulary
- Construct a list of all words in the vocabulary
- Retain only the unique words and ignore case and punctuations (recall: text pre-processing)
- From the above corpus of 24 words, we now have our vocabulary of 10 wordsย ๐
- โitโ
- โwasโ
- โtheโ
- โbestโ
- โofโ
- โtimesโ
- โworstโ
- โageโ
- โwisdomโ
- โfoolishnessโ
Step 3: Create Documentย Vectors
As we know the vocabulary has 10 words, we can use a fixed-length document vector of size 10, with one position in the vector to score eachย word.
The simplest scoring method is to mark the presence of a word as 1, if the word is present in the document, 0 otherwise
Oh yeah! thatโs simple enough; Letโs look at our document vectors now!ย ๐
โit was the best of timesโ = [1, 1, 1, 1, 1, 1, 0, 0, 0, 0]
โit was the worst of timesโ = [1, 1, 1, 0, 1, 1, 1, 0, 0, 0]
โit was the age of wisdomโ = [1, 1, 1, 0, 1, 0, 0, 1, 1, 0]
โit was the age of foolishnessโ = [1, 1, 1, 0, 1, 0, 0, 1, 0, 1]
Guess youโve already identified the issues with this approach! โ๐โโ๏ธ
- When the size of the vocabulary is large, which is the case when weโre dealing with a larger corpus, this approach would be a bit tooย tedious.
- The document vectors would be of very large length, and would be predominantly sparse, and computational efficiency is clearly suboptimal.
- As order is not preserved, context and meaning are not preserved either.
- As the Bag of Words model doesnโt consider order of words, how can we account for phrases or collection of words that occur together?
Do you remember the N-grams language model from partย 2?๐
Oh yeah, a straight forward extension of Bag-of -Words to Bag-of-N-grams helps us achieve justย that!
An N-gram is basically a collection of word tokens from a text document such that these tokens are contiguous and occur in a sequence. Bi-grams indicate n-grams of order 2 (two words), Tri-grams indicate n-grams of order 3 (three words), and soย on.
The Bag of N-Grams model is hence just an extension of the Bag of Words model so we can also leverage N-gram based features.
- This method does not take into account the relative importance of words in the text.ย ๐
Just because a word appears frequently, does it necessarily mean itโs importantย ? Well, not necessarily.
In the next section, we shall look at another metric, the TF-IDF score, which does not consider ordering of words, but aims at capturing the relative importance of words across documents in aย corpus.
Term Frequency- Inverse Document Frequency
Term Frequency- Inverse Document Frequency (TF-IDF Score) is a combination of two metricsโโโthe Term Frequency (TF) and the Inverse Document Frequency (IDF)
The idea behind TF-IDF score which is computed using the formula described below is asย follows:
โ If a word occurs frequently in a specific document, then itโs important whereas a word which occurs frequently across all documents in the corpus should be down-weighted to be able to get the words which are actually important.โ

Hereโs another widely usedย formula;

The above formula helps us calculate the TF-IDF score for term i in document j and we do it for all terms in all documents in the corpus. We therefore, get the term-document matrix of shape num_terms x num_documentsย . Hereโs anย example.
Document 1: Machine learning teaches machine how to learn
Document 2: Machine translation is my favorite subject
Document 3: Term frequency and inverse document frequency is important
Step 1: Computing f_{ij}; Frequency of term i in documentย j
For Documentย 1:

For Documentย 2:

For Documentย 3:

Step 2: Computing Normalized Term Frequency
As shown in the above formula, the f_{ij} obtained above should be divided by the total number of words in document jย .

Step 3: Compute Inverse Document Frequency (IDF) score for eachย term

Step 4: Obtain the TF-IDFย Scores
Now that weโve calculated TF_{ij} and IDF_{ij}, letโs go ahead and multiply them to get the weights w_{ij} (TF-IDF_{ij}).

Starting with raw text data, weโve successfully represented the documents in numeric form. Oh yeah! We didย it!๐
Now that we know to build numeric features from text data, as a next step, we can use these numeric representations to understand tutorials on understanding document similarity, similarity based clustering of documents in a corpus and generating topic models that are representative of latent topics in a large textย corpus.
So far, weโve looked at traditional methods in Natural language Processing. In the next part, we shall take baby steps into the realm of Deep learning forย NLP.โจ
Happy learning! Until next timeย ๐
References
Hereโs the link to the recording of theย webinar.
Extracting Features from Text Data was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.
Published via Towards AI