Web scraping & NLP

Author(s): Greg Postalian-Yrausquin

Originally published on Towards AI.

In this example, I extract data from a Wikipedia list of the most grossing movies go into each of the links and fetch the text of the movie’s article. Then I use BERTopic (which is BERT + HDBScan) to find topics in common which class these films in buckets.

This task as some particularities that made it interesting, first fetching and formatting data from HTML, then cleaning the text to get it ready for the modeling.

import pandas as pd
import numpy as np
import requests
from bs4 import BeautifulSoup
pd.options.mode.chained_assignment = None
from io import StringIO
from html.parser import HTMLParser
import re
import nltk
from nltk.corpus import stopwords
nltk.download('stopwords')
nltkstop = stopwords.words('english')
from gensim.models.doc2vec import Doc2Vec, TaggedDocument
from nltk.tokenize import word_tokenize
from nltk.stem.snowball import SnowballStemmer
nltk.download('punkt')
snow = SnowballStemmer(language='english')
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import seaborn as sns
import warnings
import seaborn as sns
from bertopic import BERTopic

Scraping a table from a website (in HTML) is as simple as:

url = 'https://en.wikipedia.org/wiki/List_of_best-selling_films_in_the_United_States'
tbls = pd.read_html(url)
tbls[0]

Now, let’s get all the urls from the website

url = 'https://en.wikipedia.org/wiki/List_of_best-selling_films_in_the_United_States'
reqs = requests.get(url)
soup = BeautifulSoup(reqs.text, 'html.parser')

urls = []
for link in soup.find_all('a'):
 row = pd.DataFrame({'Name': [link.get('title')], 'link': [link.get('href')]})
 urls.append(row)
urls = pd.concat(urls, axis=0)
urls

Since I am looking for movies and their revenue, I want only tables that have a column named “Title” and “Revenue” (or “Revenue (est.)”).

moviestbl = []
for df in tbls:
 if ('Title' in df.columns) and (('Revenue (est.)' in df.columns) or ('Revenue' in df.columns)):
 if ('Revenue (est.)' in df.columns):
 df = df[['Title','Revenue (est.)']]
 df.columns = ['Title','Revenue']
 else:
 df = df[['Title','Revenue']]
 moviestbl.append(df)
moviestbl = pd.concat(moviestbl, axis=0)
moviestbl

I’ll clean the revenue column and join the movies and hyperlink tables

moviestbl["clean_revenue"] = moviestbl["Revenue"].astype(str).str.replace('\$','').astype(str).str.replace(' ','').astype(str).str.replace(',','').apply(lambda x: x.split('[')[0])
moviestbl["clean_revenue"] = pd.to_numeric(moviestbl["clean_revenue"], errors='coerce')
moviestbl = pd.merge(left=moviestbl, right=urls, left_on='Title', right_on='Name')
moviestbl = moviestbl[["Title","clean_revenue","link"]]
moviestbl["link"] = np.where(moviestbl["link"].str.contains('https://en.wikipedia.org'),moviestbl["link"],'https://en.wikipedia.org' + moviestbl["link"])
moviestbl = moviestbl.drop_duplicates(ignore_index=True)
moviestbl = moviestbl.sort_values(['Title','clean_revenue'], ascending=False)
moviestbl = moviestbl.groupby('Title').first()
moviestbl = moviestbl.reset_index(drop=False)
moviestbl

Now it is time to extract the text (which is in the tag paragraph) from each wikipedia page. I use the same functions from my other articles for NLP that I like to apply before working with text.

class MLStripper(HTMLParser):
 def __init__(self):
 super().__init__()
 self.reset()
 self.strict = False
 self.convert_charrefs= True
 self.text = StringIO()
 def handle_data(self, d):
 self.text.write(d)
 def get_data(self):
 return self.text.getvalue()

def strip_tags(html):
 s = MLStripper()
 s.feed(html)
 return s.get_data()

def replace_words(tt, lookp_dict):
 temp = tt.split()
 res = []
 for wrd in temp:
 res.append(lookp_dict.get(wrd, wrd))
 res = ' '.join(res)
 return res

def preprepare(eingang):
 ausgang = strip_tags(eingang)
 ausgang = eingang.lower()
 ausgang = ausgang.replace(u'\xa0', u' ')
 ausgang = re.sub(r'^\s*$',' ',str(ausgang))
 ausgang = ausgang.replace('|', ' ')
 ausgang = ausgang.replace('ï', ' ')
 ausgang = ausgang.replace('»', ' ')
 ausgang = ausgang.replace('¿', '. ')
 ausgang = ausgang.replace('', ' ')
 ausgang = ausgang.replace('"', ' ')
 ausgang = ausgang.replace("'", " ")
 ausgang = ausgang.replace('?', ' ')
 ausgang = ausgang.replace('!', ' ')
 ausgang = ausgang.replace(',', ' ')
 ausgang = ausgang.replace(';', ' ')
 ausgang = ausgang.replace('.', ' ')
 ausgang = ausgang.replace("(", " ")
 ausgang = ausgang.replace(")", " ")
 ausgang = ausgang.replace("{", " ")
 ausgang = ausgang.replace("}", " ")
 ausgang = ausgang.replace("[", " ")
 ausgang = ausgang.replace("]", " ")
 ausgang = ausgang.replace("~", " ")
 ausgang = ausgang.replace("@", " ")
 ausgang = ausgang.replace("#", " ")
 ausgang = ausgang.replace("$", " ")
 ausgang = ausgang.replace("%", " ")
 ausgang = ausgang.replace("^", " ")
 ausgang = ausgang.replace("&", " ")
 ausgang = ausgang.replace("*", " ")
 ausgang = ausgang.replace("<", " ")
 ausgang = ausgang.replace(">", " ")
 ausgang = ausgang.replace("/", " ")
 ausgang = ausgang.replace("\\", " ")
 ausgang = ausgang.replace("`", " ")
 ausgang = ausgang.replace("+", " ")
 ausgang = ausgang.replace("=", " ")
 ausgang = ausgang.replace("_", " ")
 ausgang = ausgang.replace("-", " ")
 ausgang = ausgang.replace(':', ' ')
 ausgang = ausgang.replace('\n', ' ').replace('\r', ' ')
 ausgang = ausgang.replace(" +", " ")
 ausgang = ausgang.replace(" +", " ")
 ausgang = ausgang.replace('?', ' ')
 ausgang = re.sub('[^a-zA-Z]', ' ', ausgang)
 ausgang = re.sub(' +', ' ', ausgang)
 ausgang = re.sub('\ +', ' ', ausgang)
 ausgang = re.sub(r'\s([?.!"](?:\s|$))', r'\1', ausgang)
 return ausgang

This small function is the core of the web scrapping process with beautiful soup:

def scraptext(txt):
 reqs = requests.get(txt)
 soup = BeautifulSoup(reqs.text, 'html.parser')
 ps = []
 for par in soup.find_all('p'):
 ps.append(par.text)
 ps = ' '.join(ps)
 return ps

Now, let’s apply them

moviestbl["text"] = moviestbl["link"].apply(lambda x: scraptext(x))
moviestbl["text"] = moviestbl["text"].apply(lambda x: preprepare(x))
moviestbl["text"] = moviestbl["text"].apply(lambda x: ' '.join([word for word in x.split() if word not in (nltkstop)]))
moviestbl = moviestbl[moviestbl["text"] != " "]
moviestbl = moviestbl[moviestbl["text"] != ""]
moviestbl = moviestbl.dropna()

moviestbl

Now the data is ready for NLP, I will use BERT, in specific Bertopic, which is a package specifically designed for topic modeling

model = BERTopic(verbose=True,embedding_model='bert-base-uncased', min_topic_size= 3)
headline_topics, _ = model.fit_transform(moviestbl["text"])

As simple as that. Now I assign the topics to the text and explore the topic information.

moviestbl["topic"] = headline_topics
topicinfo = model.get_topic_info()
topicinfo

Some more useful information about a topic are the words attached to it and their score, and at the same time let’s list the movies assigned to it

topic = 0
print("topic: " + str(topic))
print(model.get_topic(topic))
print(moviestbl[moviestbl['topic']==topic]['Title'])
topic = 1
print("topic: " + str(topic))
print(model.get_topic(topic))
print(moviestbl[moviestbl['topic']==topic]['Title'])

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI