Building a Spam Detector Using Python’s NTLK Package

Author(s): Bindhu Balu

Originally published on Towards AI.

NTLK — Natural Language ToolKit

In this part, we will go through an end to end walkthrough of building a very simple text classifier in Python 3.

Our goal is to build a predictive model that will determine whether a text message is a spam or ham.

Code Location https://github.com/BindhuVinodh/Spam-Detector

/Spam-Detector

# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load in import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)# Input data files are available in the "../input/" directory.
# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directoryimport os
print(os.listdir("../input"))# Any results you write to the current directory are saved as output.['SMSSpamCollection', 'readme']

Get the Data

In [2]:

message = [line.rstrip() for line in open('../input/SMSSpamCollection')]
print(len(message))5574

In [3]:

for message_no,message in enumerate(message[:10]):
 print(message_no,message)
 print('\n')0 ham	Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...
1 ham	Ok lar... Joking wif u oni...
2 spam	Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's
3 ham	U dun say so early hor... U c already then say...
4 ham	Nah I don't think he goes to usf, he lives around here though
5 spam	FreeMsg Hey there darling it's been 3 week's now and no word back! I'd like some fun you up for it still? Tb ok! XxX std chgs to send, £1.50 to rcv
6 ham	Even my brother is not like to speak with me. They treat me like aids patent.
7 ham	As per your request 'Melle Melle (Oru Minnaminunginte Nurungu Vettam)' has been set as your callertune for all Callers. Press *9 to copy your friends Callertune
8 spam	WINNER!! As a valued network customer you have been selected to receivea £900 prize reward! To claim call 09061701461. Claim code KL341. Valid 12 hours only.
9 spam	Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030

In [4]:

import pandas as pd

In [5]:

message=pd.read_csv('../input/SMSSpamCollection',sep='\t',names=["labels","message"])
message.head()

Out[5]:

labelsmessage0hamGo until jurong point, crazy.. Available only …1hamOk lar… Joking wif u oni…2spamFree entry in 2 a wkly comp to win FA Cup fina…3hamU dun say so early hor… U c already then say…4hamNah I don’t think he goes to usf, he lives aro…

Exploratory Data Analysis

In [6]:

message.describe()

Out[6]:

labelsmessagecount55725572unique25169tophamSorry, I’ll call laterfreq482530

In [7]:

message.groupby('labels').describe()

Out[7]:

messagecountuniquetopfreqlabelsham48254516Sorry, I’ll call later30spam747653Please call our customer service representativ…4

As we continue our analysis we want to start thinking about the features we are going to be using. This goes along with the general idea of feature engineering. The better your domain knowledge on the data, the better your ability to engineer more features from it. Feature engineering is a very large part of spam detection in general. I encourage you to read up on the topic!

Let’s make a new column to detect how long the text messages are:

In [8]:

message['length']=message['message'].apply(len)
message.head()

Out[8]:

labelsmessagelength0hamGo until jurong point, crazy.. Available only …1111hamOk lar… Joking wif u oni…292spamFree entry in 2 a wkly comp to win FA Cup fina…1553hamU dun say so early hor… U c already then say…494hamNah I don’t think he goes to usf, he lives aro…61

Data Visualization

Let’s visualize this! Let’s do the imports:

In [9]:

import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

In [10]:

message['length'].plot(bins=50,kind='hist')

Out[10]:

<matplotlib.axes._subplots.AxesSubplot at 0x7fdf3c7d92e8>

In [11]:

message.length.describe()

Out[11]:

count 5572.000000
mean 80.489950
std 59.942907
min 2.000000
25% 36.000000
50% 62.000000
75% 122.000000
max 910.000000
Name: length, dtype: float64

Woah! 910 characters, let’s use masking to find this message:

In [12]:

message[message['length']==910]['message'].iloc[0]

Out[12]:

"For me the love should start with attraction.i should feel that I need her every time around me.she should be the first thing which comes in my thoughts.I would start the day and end it with her.she should be there every time I dream.love will be then when my every breath has her name.my life should happen around her.my life will be named to her.I would cry for her.will give all my happiness and take all her sorrows.I will be ready to fight with anyone for her.I will be in love when I will be doing the craziest things for her.love will be when I don't have to proove anyone that my girl is the most beautiful lady on the whole planet.I will always be singing praises for her.love will be when I start up making chicken curry and end up makiing sambar.life will be the most beautiful then.will get every morning and thank god for the day because she is with me.I would like to say a lot..will tell later.."

Looks like we have some sort of Romeo sending texts! But let’s focus back on the idea of trying to see if message length is a distinguishing feature between ham and spam

Text Pre-processing

Our main issue with our data is that it is all in text format (strings). The classification algorithms that we’ve learned about so far will need some sort of numerical feature vector in order to perform the classification task. There are actually many methods to convert a corpus into a vector format. The simplest is the bag-of-words approach, where each unique word in a text will be represented by one number.

In this section, we’ll convert the raw messages (sequence of characters) into vectors (sequences of numbers).

As a first step, let’s write a function that will split a message into its individual words and return a list. We’ll also remove very common words, (‘the’, ‘a’, etc..). To do this we will take advantage of the NLTK library. It’s pretty much the standard library in Python for processing text and has a lot of useful features. We’ll only use some of the basic ones here.

Let’s create a function that will process the string in the message column, then we can just use apply() in pandas do process all the text in the DataFrame.

First removing punctuation. We can just take advantage of Python’s built-in string library to get a quick list of all the possible punctuation:

In [13]:

import string
mess = 'sample message!...'
nopunc=[char for char in mess if char not in string.punctuation]
nopunc=''.join(nopunc)
print(nopunc)sample message

In [14]:

from nltk.corpus import stopwords
stopwords.words('english')[0:10]

Out[14]:

['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're"]

In [15]:

nopunc.split()

Out[15]:

['sample', 'message']

In [16]:

clean_mess=[word for word in nopunc.split() if word.lower() not in stopwords.words('english')]

In [17]:

clean_mess

Out[17]:

['sample', 'message']

Now let’s put both of these together in a function to apply it to our DataFrame later on:

In [18]:

def text_process(mess):
 nopunc =[char for char in mess if char not in string.punctuation]
 nopunc=''.join(nopunc)
 return [word for word in nopunc.split() if word.lower() not in stopwords.words('english')]

Here is the original DataFrame again:

In [19]:

message.head()

Out[19]:

labelsmessagelength0hamGo until jurong point, crazy.. Available only …1111hamOk lar… Joking wif u oni…292spamFree entry in 2 a wkly comp to win FA Cup fina…1553hamU dun say so early hor… U c already then say…494hamNah I don’t think he goes to usf, he lives aro…61

Now let’s “tokenize” these messages. Tokenization is just the term used to describe the process of converting the normal text strings in to a list of tokens (words that we actually want).

Let’s see an example output on on column:

Note: We may get some warnings or errors for symbols we didn’t account for or that weren’t in Unicode (like a British pound symbol)

In [20]:

message['message'].head(5).apply(text_process)

Out[20]:

0 [Go, jurong, point, crazy, Available, bugis, n...
1 [Ok, lar, Joking, wif, u, oni]
2 [Free, entry, 2, wkly, comp, win, FA, Cup, fin...
3 [U, dun, say, early, hor, U, c, already, say]
4 [Nah, dont, think, goes, usf, lives, around, t...
Name: message, dtype: object

In [21]:

message.head()

Out[21]:

labelsmessagelength0hamGo until jurong point, crazy.. Available only …1111hamOk lar… Joking wif u oni…292spamFree entry in 2 a wkly comp to win FA Cup fina…1553hamU dun say so early hor… U c already then say…494hamNah I don’t think he goes to usf, he lives aro…61

Continuing Normalization

There are a lot of ways to continue normalizing this text. Such as Stemming or distinguishing by part of speech.

NLTK has lots of built-in tools and great documentation on a lot of these methods. Sometimes they don’t work well for text-messages due to the way a lot of people tend to use abbreviations or shorthand, For example:

'Nah dawg, IDK! Wut time u headin to da club?'

versus

'No dog, I don't know! What time are you heading to the club?'

Some text normalization methods will have trouble with this type of shorthand and so I’ll leave you to explore those more advanced methods through the NLTK book online.

For now, we will just focus on using what we have to convert our list of words to an actual vector that SciKit-Learn can use.

Vectorization

Currently, we have the messages as lists of tokens (also known as lemmas) and now we need to convert each of those messages into a vector the SciKit Learn’s algorithm models can work with.

Now we’ll convert each message, represented as a list of tokens (lemmas) above, into a vector that machine learning models can understand.

We’ll do that in three steps using the bag-of-words model:

1. Count how many times does a word occur in each message (Known as term frequency)
2. Weigh the counts, so that frequent tokens get lower weight (inverse document frequency)
3. Normalize the vectors to unit length, to abstract from the original text length (L2 norm)

Let’s begin the first step:

Each vector will have as many dimensions as there are unique words in the SMS corpus. We will first use SciKit Learn’s CountVectorizer. This model will convert a collection of text documents to a matrix of token counts.

We can imagine this as a 2-Dimensional matrix. Where the 1-dimension is the entire vocabulary (1 row per word) and the other dimension are the actual documents, in this case, a column per text message.

For example:

<table border = “1“>

Message 1 Message 2 … Message N </tr>

Word 1 Count01…0 </tr>

Word 2 Count00…0 </tr>

… 12…0 </tr>

Word N Count 01…1 </tr> </table>

Since there are so many messages, we can expect a lot of zero counts for the presence of that word in that document. Because of this, SciKit Learn will output a Sparse Matrix.

In [22]:

from sklearn.feature_extraction.text import CountVectorizer

Let’s take one text message and get its bag-of-words counts as a vector, putting to use our new bow_transformer:

In [23]:

bow_transformer = CountVectorizer(analyzer=text_process).fit(message['message'])
print(len(bow_transformer.vocabulary_))11425

In [24]:

message4=message['message'][3]
print(message4)U dun say so early hor... U c already then say...

Now let’s see its vector representation:

In [25]:

bow4=bow_transformer.transform([message4])
print(bow4)
print(bow4.shape)(0, 4068)	2
 (0, 4629)	1
 (0, 5261)	1
 (0, 6204)	1
 (0, 6222)	1
 (0, 7186)	1
 (0, 9554)	2
(1, 11425)

This means that there are seven unique words in message number 4 (after removing common stop words). Two of them appear twice, the rest only once. Let’s go ahead and check and confirm which ones appear twice:

In [26]:

print(bow_transformer.get_feature_names()[4073])
print(bow_transformer.get_feature_names()[9570])UIN
schedule

Now we can use .transform on our Bag-of-Words (bow) transformed object and transform the entire DataFrame of messages. Let’s go ahead and check out how the bag-of-words counts for the entire SMS corpus is a large, sparse matrix:

In [27]:

messages_bow = bow_transformer.transform(message['message'])

In [28]:

print('Shape of Sparse Matrix: ',messages_bow.shape)
print('Amount of non-zero occurences:',messages_bow.nnz)Shape of Sparse Matrix: (5572, 11425)
Amount of non-zero occurences: 50548

In [29]:

sparsity =(100.0 * messages_bow.nnz/(messages_bow.shape[0]*messages_bow.shape[1]))
print('sparsity:{}'.format(round(sparsity)))sparsity:0

After the counting, the term weighting and normalization can be done with TF-IDF, using scikit-learn’s TfidfTransformer.

So what is TF-IDF?

TF-IDF stands for term frequency-inverse document frequency, and the tf-idf weight is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus. Variations of the tf-idf weighting scheme are often used by search engines as a central tool in scoring and ranking a document’s relevance given a user query.

One of the simplest ranking functions is computed by summing the tf-idf for each query term; many more sophisticated ranking functions are variants of this simple model.

Typically, the tf-idf weight is composed by two terms: the first computes the normalized Term Frequency (TF), aka. the number of times a word appears in a document, divided by the total number of words in that document; the second term is the Inverse Document Frequency (IDF), computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears.

TF: Term Frequency, which measures how frequently a term occurs in a document. Since every document is different in length, it is possible that a term would appear much more time in long documents than shorter ones. Thus, the term frequency is often divided by the document length (aka. the total number of terms in the document) as a way of normalization:

TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document).

IDF: Inverse Document Frequency, which measures how important a term is. While computing TF, all terms are considered equally important. However, it is known that certain terms, such as “is”, “of”, and “that”, may appear a lot of times but have little importance. Thus we need to weigh down the frequent terms while scaling up the rare ones, by computing the following:

IDF(t) = log_e(Total number of documents / Number of documents with term t in it).

See below for a simple example.

Example:

Consider a document containing 100 words wherein the word cat appears 3 times.

The term frequency (i.e., tf) for cat is then (3 / 100) = 0.03. Now, assume we have 10 million documents and the word cat appears in one thousand of these. Then, the inverse document frequency (i.e., idf) is calculated as log(10,000,000 / 1,000) = 4. Thus, the Tf-idf weight is the product of these quantities: 0.03 * 4 = 0.12.

Let’s go ahead and see how we can do this in SciKit Learn:

In [30]:

from sklearn.feature_extraction.text import TfidfTransformer
tfidf_transformer=TfidfTransformer().fit(messages_bow)
tfidf4 = tfidf_transformer.transform(bow4)
print(tfidf4)(0, 9554)	0.5385626262927564
 (0, 7186)	0.4389365653379857
 (0, 6222)	0.3187216892949149
 (0, 6204)	0.29953799723697416
 (0, 5261)	0.29729957405868723
 (0, 4629)	0.26619801906087187
 (0, 4068)	0.40832589933384067

We’ll go ahead and check what is the IDF (inverse document frequency) of the word "u" and of word "university"?

In [31]:

print(tfidf_transformer.idf_[bow_transformer.vocabulary_['u']])
print(tfidf_transformer.idf_[bow_transformer.vocabulary_['university']])3.2800524267409408
8.527076498901426

In [32]:

messages_tfidf=tfidf_transformer.transform(messages_bow)
print(messages_tfidf.shape)(5572, 11425)

There are many ways the data can be preprocessed and vectorized. These steps involve feature engineering and building a “pipeline”. I encourage you to check out SciKit Learn’s documentation on dealing with text data as well as the expansive collection of available papers and books on the general topic of NLP.

Training a model

With messages represented as vectors, we can finally train our spam/ham classifier. Now we can actually use almost any sort of classification algorithm. For a variety of reasons, the Naive Bayes classifier algorithm is a good choice.

In [33]:

from sklearn.naive_bayes import MultinomialNB
spam_detect_model = MultinomialNB().fit(messages_tfidf,message['labels'])

In [34]:

print('predicted:',spam_detect_model.predict(tfidf4)[0])
print('expected:',message.labels[3])predicted: ham
expected: ham

Fantastic! We’ve developed a model that can attempt to predict spam vs ham classification!

Part 6: Model Evaluation

Now we want to determine how well our model will do overall on the entire dataset. Let’s begin by getting all the predictions:

In [35]:

all_predictions = spam_detect_model.predict(messages_tfidf)
print(all_predictions)['ham' 'ham' 'spam' ... 'ham' 'ham' 'ham']

We can use SciKit Learn’s built-in classification report, which returns precision, recall, f1-score, and a column for support (meaning how many cases supported that classification). Check out the links for more detailed info on each of these metrics and the figure below:

In [36]:

from sklearn.metrics import classification_report,confusion_matrix
print(classification_report(message['labels'],all_predictions))
print(confusion_matrix(message['labels'],all_predictions))precision recall f1-score support ham 0.98 1.00 0.99 4825
 spam 1.00 0.85 0.92 747avg / total 0.98 0.98 0.98 5572[[4825 0]
 [ 115 632]]

There are quite a few possible metrics for evaluating model performance. Which one is the most important depends on the task and the business effects of decisions based on the model? For example, the cost of mispredicting “spam” as “ham” is probably much lower than mispredicting “ham” as “spam”.

In the above “evaluation”, we evaluated accuracy on the same data we used for training. You should never actually evaluate on the same dataset you train on!

Such evaluation tells us nothing about the true predictive power of our model. If we simply remembered each example during training, the accuracy of training data would trivially be 100%, even though we wouldn’t be able to classify any new messages.

A proper way is to split the data into a training/test set, where the model only ever sees the training data during its model fitting and parameter tuning. The test data is never used in any way. This is then our final evaluation of test data is representative of true predictive performance.

Train Test Split

In [37]:

from sklearn.model_selection import train_test_split
msg_train,msg_test,label_train,label_test = train_test_split(message['message'],message['labels'],test_size=0.2)

In [38]:

print(len(msg_train),len(msg_test),len(label_train),len(label_test))4457 1115 4457 1115

The test size is 20% of the entire dataset (1115 messages out of total 5572), and the training is the rest (4457 out of 5572). Note the default split would have been 30/70.

Creating a Data Pipeline

Let’s run our model again and then predict off the test set. We will use SciKit Learn’s pipeline capabilities to store a pipeline of the workflow. This will allow us to set up all the transformations that we will do to the data for future use. Let’s see an example of how it works:

In [39]:

from sklearn.pipeline import Pipeline
pipeline = Pipeline([
 ( 'bow',CountVectorizer(analyzer=text_process)),
 ('tfidf',TfidfTransformer()),
 ('classifier',MultinomialNB()),
])

In [40]:

pipeline.fit(msg_train,label_train)

Out[40]:

Pipeline(memory=None,
 steps=[('bow', CountVectorizer(analyzer=<function text_process at 0x7fdf3393f510>,
 binary=False, decode_error='strict', dtype=<class 'numpy.int64'>,
 encoding='utf-8', input='content', lowercase=True, max_df=1.0,
 max_features=None, min_df=1, ngram_range=(1, 1), preprocessor=No...f=False, use_idf=True)), ('classifier', MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))])

In [41]:

predictions = pipeline.predict(msg_test)

In [42]:

print(classification_report(predictions,label_test))precision recall f1-score support ham 1.00 0.96 0.98 1003
 spam 0.74 1.00 0.85 112avg / total 0.97 0.97 0.97 1115

The test size is 20% of the entire dataset (1115 messages out of total 5572), and the training is the rest (4457 out of 5572). Note the default split would have been 30/70.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI