Building a Text Summarizer with Transformer

Author(s): SHARON ZACHARIA

Originally published on Towards AI.

Can machines understand human language? What if we could interact with them the same way we do with other humans? These questions are addressed by the field of Natural Language processing, which allows machines to mimic human comprehension and usage of natural language. Early foundations of NLP were established by statistical and rule-based models like the Bag of Words (BoW). Even while traditional methods are straightforward, they are nevertheless widely used today and are often contrasted with more sophisticated models based on Transformers. In this article, we will discuss what BoW is and how Transformers revolutionized the field of NLP over time. We will then implement BoW and a Transformer based text summarizer model using T5 model.

Extractive Summarization extracts key sentences, phrases, or sections directly from the original text to create a summary. It doesn’t generate new sentences but identifies the most important pieces of information based on factors like sentence importance, relevance, and position in the text.

Abstractive Summarization: generates a summary by understanding the original content and then rephrasing or rewriting it in a more concise form. The model generates new sentences that may not appear in the original text but capture the main ideas. It is typically more flexible and capable of producing more human-like summaries

Bag of Words

BoW is one of the foundational models in NLP for text representation. It represents text in numerical format, considering each document as a collection of words. BoW mainly focuses on word frequency, ignoring grammar and word order. It is one of the widely used technique in NLP despite its simplicity. Bag of Words Representation mainly involves a vocabulary of words and Measure of presence of each word. Consider two sentences as follows:

• Sentence 1 : “I Love Cats”
• Sentence 2 : “I Love Dogs”

After tokenisation process the sentence is split into individual words(tokens) this will be as follows.

• Sentence 1 : [”I”, ”Love”, ”Cats”]
• Sentence 2 : [”I”, ”Love”, ”Dogs”]

The resulting vocabulary would be as follows:

[”I” , ”Love” , ”Cats” , ”Dogs”]

The resulting vectors will be as follows :

• Sentence 1 : [1, 1, 1, 0]. ”I” , ”Love” , ”Cats” appears once (1) and ”Dogs”is not present (0) in the first sentence.
• Sentence 2 : [1, 1, 0 ,1]. ”I” , ”Love”, ”Dogs” appears once (1) and ”Cats” is not present (0) in the Second sentence.

Similarity Matrix

A similarity matrix is a mathematical representation of how sentences are similar to each other. Here the similarity of sentences are calculated using cosine similarity. Cosine similarity measures the cosine of the angle between two vectors.

Transformer Architecture

The transformer architecture was introduced in the paper Attention is all you need (Vaswani et al. 2017) which revolutionized the field of Natural Language Processing and Machine Learning by addressing the limitations of CNNs (Convolutional Neural Networks)and RNNs (Recurrent Neural Networks). It uses a self-attention mechanism to analyse sequential input, allowing for parallel computation and reaching cutting-edge results in tasks like question-answering, summarization, and text translation.

The Encoder is the fundamental component of the Transformer architecture, which transforms input tokens into contextualized representations instead of processing tokens independently. Encoder captures the context of each token in the entire sequence. Using embedding layers, the encoder first transforms input tokens into vectors. The tokens’ semantic meaning is captured by these embeddings, which then translate them into numerical vectors.

The Decoder is in charge of producing the output sequence one step at a time using the encoded input sequence and its own outputs that have already been produced. A masked multi-head self-attention mechanism and a multi-head cross-attention mechanism are the two primary sublayers that make up each of its several layers. A feedforward neural network comes next. The self-attention method enables the decoder to predict the next token by taking into account just the previously created tokens. The decoder can concentrate on pertinent segments of the input sequence by paying attention to the encoder’s output representations with the help of cross-attention method.

Self Attention Mechanism in Transformer architecture creates three representation of the tokens in order to weight the importance of each word in the sequence relative to one another. The three representations include Queries (Q) , Keys (K) and Values (V) which are obtained by multiplying the input embeddings with learned weight matrices. To find how much attention each token in the sequence should pay to the others, the mechanism calculates a similarity score between its query vector and the key vectors of all tokens. These similarity scores are scaled by the square root of the key vectors’ dimensions. A softmax function is then used to normalize the scores, creating attention weights. After applying the weights to the value vectors, a weighted total is created that considers each token’s context in respect to the others.

Since we have discussed important concepts both in BoW and Transformers, let’s try implementing these models. As discussed earlier, we will be implementing these two models to summarize textual data. The data set used is cnn/DailyMail which is a widely used dataset for NLP tasks. The dataset can be found here

Bag of Words

We will be using the below-mentioned libraries to implement the model.

import nltk
import re 
from nltk.corpus import stopwords
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import cosine_similarity

We will begin by implementing text preprocessing steps which helps to convert the entire text to lowercase, ensuring consistency and avoiding case-sensitive issues. Non-alphanumeric characters are removed using regular expressions, leaving only relevant words and spaces. The function then splits the text into tokens and filters out common stop words, using a predefined list of English stop words.

# Download NLTK stopwords
nltk.download("stopwords")
nltk.download('punkt_tab')
stop_words = set(stopwords.words("english")

def preprocess_text(text):
 text = text.lower()
 text = re.sub(r'\W', ' ', text)
 # Removing stopwords
 tokens = text.split()
 tokens = [word for word in tokens if word not in stop_words]
 return ' '.join(tokens)

Now we will define a function that takes text as input and gives a summary of it. The input text is split into sentences using nltk.sent_tokenize() function, and each sentence is preprocessed using the above defined preprocess function. CountVectorizer() is used to vectorize the sentences, and a similarity score is calculated using cosine_similarity(). The top ranked sentences are selected and returned, here the num_sentences holds the value for the number of sentences to be returned as summary.

def summarize_with_bow(article_text, num_sentences=2):
 # Split article into sentences and preprocess it
 sentences = nltk.sent_tokenize(article_text)
 processed_sentences = [preprocess_text(sentence) for sentence in sentences]

 # Vectorize sentences using Bag of Words
 vectorizer = CountVectorizer()
 sentence_vectors = vectorizer.fit_transform(processed_sentences)

 # Compute similarity score
 similarity_matrix = cosine_similarity(sentence_vectors)
 sentence_scores = similarity_matrix.sum(axis=1)
 ranked_sentences = np.argsort(sentence_scores)[::-1]

 # Select top sentences
 selected_sentences = [sentences[i] for i in ranked_sentences[:num_sentences]]
 summary = " ".join(selected_sentences)
 return summary

The original article and the generated summary are as follows.

Original Article: Fenerbahce have called for the suspension of the Turkish
championship following the gun attack on their team bus on Saturday.The bus 
came under armed attack as it drove to the airport following an away match at 
Caykur Rizespor in Turkey's Super Lig. Fener said on their official website 
the bus driver was wounded in the attack and taken to hospital and there was 
no mention of anyinjuries to anyone else. Marks can be seen on the windscreen 
of the Fenerbahce team bus after the attack .............................

Generated Summary: Fenerbahce have called for the suspension of the Turkish 
championship following the gun attack on their team bus on Saturday.Marks can
be seen on the windscreen of the Fenerbahce team bus after the attack .

As we can see, BoW method uses extractive summarization technique to generate a summary which ignores meaning and context of the text and generates summaries based on occurrence of each word, which it finds relevant. Now we will see how transformer based models generate abstractive summaries from the given text.

Text To Text Transfer Transformer (T5)

T5 model is a transformer based model from Google, it treats every NLP task in a text-to-text format allowing both input and output to be in text format. T5 is built on an encoder-decoder architecture, in which the encoder processes the input and decoder generates new content. We will begin by importing the necessary libraries, the use of each of these will be discussed as we progress through the code.

#Import Libraries
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, DataCollatorForSeq2Seq, Seq2SeqTrainingArguments, Seq2SeqTrainer
import evaluate
import numpy as np

As discussed earlier, we will be using the CNN/Daily Mail dataset for training and testing the model. Let’s load the dataset ! A subset of the dataset is created by sampling the first 30,000 instances from the training set to optimize for computational efficiency. This sampled dataset is then split into training and test sets, with 80% used for training and 20% for testing. AutoTokenizer from Transformer module is used for tokenising the articles and highlights. It allows us to select the best tokenizer for the model without having to know any specifics about the model.

cnn_dailymail = load_dataset("cnn_dailymail", "1.0.0")
sampled_dataset = cnn_dailymail["train"].select(range(30000))
cnn_dailymail = sampled_dataset.train_test_split(test_size=0.2)
checkpoint = "t5-small"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

The preprocess_function prepares the dataset for model training by prefixing each article with “summarize:” to guide the model’s summarization task. It tokenizes the input articles and the corresponding summaries while applying truncation and padding to ensure the sequences are of appropriate length. The tokenized summaries are added as labels to the model inputs, ready for training.

def preprocess_function(examples):
 inputs = ["summarize: " + doc for doc in examples["article"]]
 model_inputs = tokenizer(inputs, max_length=1024, truncation=True)
 labels = tokenizer(text_target=examples["highlights"], max_length=128, truncation=True,padding=True)
 model_inputs["labels"] = labels["input_ids"]
 return model_inputs

The dataset is tokenized using the preprocess_function with batching enabled. A DataCollatorForSeq2Seq() is created to handle padding and batching during training, using the T5 tokenizer and model checkpoint. The ROUGE metric is loaded to evaluate model performance on summarization tasks. Finally, the T5 model is loaded from the pre-trained “t5-small” checkpoint, ready for fine-tuning on the dataset.

tokenized_cnn_dailymail = cnn_dailymail.map(preprocess_function, batched=True)
data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=checkpoint)
rouge = evaluate.load("rouge")
model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)

The compute_metrics function is used to evaluate the performance of a text generation model, It takes eval_pred as input, which contains the model’s predicted outputs and the true labels. The predictions and labels are then decoded from token IDs to human-readable text using the tokenizer.batch_decode function, skipping any special tokens like padding. Any padding tokens in the labels are replaced with the tokenizer’s padding token ID to ensure proper handling.

def compute_metrics(eval_pred):
 predictions, labels = eval_pred
 decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)
 labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
 decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

 result = rouge.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
 prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in predictions]
 result["gen_len"] = np.mean(prediction_lens)

 print("Sample Predictions:", decoded_preds[:3]) #predictions
 return {k: round(v, 4) for k, v in result.items()}

Training arguments for the model is defined below,

• output_dir: This specifies the path to the directory where the trained model and checkpoints will be saved during training.
• evaluation_strategy=”epoch”: The model will be evaluated at the end of each training epoch
• learning_rate=2e-5: This sets the learning rate for the optimizer, which controls how much the model’s weights are adjusted during training.
• per_device_train_batch_size=8: This determines how many examples will be processed in parallel for each training step
• per_device_eval_batch_size=8: determines the evaluation batch size.
• save_total_limit=3: This sets the maximum number of checkpoints to keep during training.
• num_train_epochs=3: This specifies how many times the model will loop over the entire training dataset.
• fp16=True: This enables 16-bit floating point precision during training, which helps speed up training and reduce memory usage,

training_args = Seq2SeqTrainingArguments(
 output_dir="<Path to directory>",
 evaluation_strategy="epoch",
 learning_rate=2e-5,
 per_device_train_batch_size=8,
 per_device_eval_batch_size=8,
 weight_decay=0.01,
 save_total_limit=3,
 num_train_epochs=3,
 predict_with_generate=True,
 fp16=True,
)

Here, the trainer is initialized with the required values. training_args is passed to trainer along with the preprocessed dataset for training and evaluation. The trainer.train() starts the model’s training.

trainer = Seq2SeqTrainer(
 model=model,
 args=training_args,
 train_dataset=tokenized_cnn_dailymail["train"],
 eval_dataset=tokenized_cnn_dailymail["test"],
 tokenizer=tokenizer,
 data_collator=data_collator,
 compute_metrics=compute_metrics,)

trainer.train()

The model starts generating summaries after each epoch, where the content of the summary changes, since the ability of the model to generate a more meaningful summary improves after each epoch. The quality of the summary increases with more training data and the number of epochs. Once the model is trained, we can infer the model to generate summaries on the text it has never seen. This can be achieved as follows.

model_name = "<Patht to model checkpoint directory>" #loading the model checkpoint
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

We will use the below function to generate the summaries. This fucntion tokenizes the given article, generates a summary and decodes the summary into readable text, and returns it. The values for top_k , top_p , temperature and num_beams decides the quality of the generated summary.

def summerize_with_T5(article_text):
 
 inputs = tokenizer(article_text, return_tensors="pt", max_length=1024, truncation=True)
 outputs = model.generate(
 **inputs,
 max_length=1024,
 early_stopping=True,
 temperature=0.8,
 top_k=70,
 top_p=0.8,
 num_beams=4,
 )
 summary = tokenizer.decode(outputs[0], skip_special_tokens=True)
 return summary

We have seen that Transformer-based models generate summaries which are meaningful and more context aware than that of traditional approaches, which make these models more powerful in Natural Language Processing tasks.

Useful Links

• Attention Is All You Need
• How do transformers work
• GitHub Repository
• Visualizing Transformers and Attention

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI