Scraping Your Medium Stories

Author(s): Joaquin de Castro

Originally published on Towards AI.

Data Mining, Programming

While learning about and exploring the power of web scraping

In the “good old days” Medium allowed us to have custom websites for our publications. Unfortunately, this functionality has been recently deprecated.

Nonetheless, as developers, we are resourceful and not so easily discouraged.

We can use web scraping to get details on each of our Medium stories (title, description, link, feature image, and even tags!) and display them on a separate website.

This process will be divided into two articles. The first part will focus solely on scraping the articles, where we can discover the power of web scraping. In the second article, we will use our scraped data to add and update entries to a database and automate and even schedule the scraping process.

Web Scraping

Web scraping is the process of extracting data from a website. We can use it to ‘scrape’ social media posts, news articles, and in our case, Medium stories. To help us out, we can use a couple of libraries, including BeautifulSoup and requests. With these, it really only takes three to four lines to scrape almost anything:

url='https://medium.com'
response = requests.get(url)
content = BeautifulSoup(response.content, 'html.parser')
some_tag = content.find_all('tag')

The first line just defines the URL which we will be getting data from. The second line actually accesses the URL, the same way a human would paste a URL in a search engine. The third line parses the Response Object returned from the second line and gives us HTML content we can actually read. And finally, the fourth line searches the content for a specific HTML tag.

Of course, a web scraper is rarely ever just four lines. We could want to search for different tags scattered throughout the web page, and format them so that they are human-readable. But the entire process remains the same.

Getting our hands dirty with code

So without any more delay, let’s get into the actual code. Again, our goal is to scrape an author’s Medium stories and provide the relevant information on each.

The URL

As we saw above, we need to have an actual web page where we can get all that juicy data. We could try our Medium profile page (like https://medium.com/@joaquindecastro), but the information there is buried in messy source code, which can be a nuisance to parse. Instead, we can use the RSS feed provided by Medium for each author (like https://medium.com/feed/@joaquindecastro).

When we open this page, we can see that the information is already laid out in XML tags. Not only is this much easier to scrape, but we can avoid inspecting any source code entirely!

url='https://medium.com/feed/@joaquindecastro'

Parsing With Requests and BeautifulSoup

Next, we can simply follow the process outlined above to get the content of the URL:

response = requests.get(url)
content = BeautifulSoup(response.content, 'html.parser')

Now, before we get into retrieving information, notice some texts are nested in XML character data blocks (<![CDATA[]]>). Let’s remove this since we will be parsing the feed as HTML.

content = str(content).replace('<![CDATA[','').replace(']]>','')
content = BeautifulSoup(content, 'html.parser')

U+1F4A1 Remember to convert the content variable to a string first and back to a BeautifulSoup object, without doing so we will run into attribute and type errors

Get A List of Recent Articles

Looking at the RSS feed, we see that information on each article is found in an <item> tag.

A snippet of my RSS feed<item>
<title>
<![CDATA[ The Powers of Two: Why Is 1 + 2 + 4 + 8 + … = -1 ]]>
</title>
<description>
<![CDATA[ <div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/cantors-paradise/the-powers-of-two-why-is-1-2-4-8-1-19d8f00be228?source=rss-46b79b6c143b------2"><img src="https://cdn-images-1.medium.com/max/2048/1*geeybYiAeCd1vFkKuO75lA@2x.jpeg" width="2048"></a></p><p class="medium-feed-snippet">On calculating infinite divergent series sums</p><p class="medium-feed-link"><a href="https://medium.com/cantors-paradise/the-powers-of-two-why-is-1-2-4-8-1-19d8f00be228?source=rss-46b79b6c143b------2">Continue reading on Cantor’s Paradise »</a></p></div> ]]>
</description>
<link>https://medium.com/cantors-paradise/the-powers-of-two-why-is-1-2-4-8-1-19d8f00be228?source=rss-46b79b6c143b------2</link>
<guid isPermaLink="false">https://medium.com/p/19d8f00be228</guid>
<category>
<![CDATA[ science ]]>
</category>
<category>
<![CDATA[ infinity ]]>
</category>
<category>
<![CDATA[ math ]]>
</category>
<category>
<![CDATA[ education ]]>
</category>
<category>
<![CDATA[ numbers ]]>
</category>
<dc:creator>
<![CDATA[ Joaquin de Castro ]]>
</dc:creator>
<pubDate>Wed, 01 Jul 2020 08:36:11 GMT</pubDate>
<atom:updated>2020-07-01T10:45:28.172Z</atom:updated>
</item>

Since we want to get information for each article, we should iterate over all these item tags.

To get an ‘iterable’ object, we can use BeautifulSoup’s find_all attribute like so:

articles = content.find_all('item')

Then, we can loop over this:

for a in articles:
 # GET TITLE
 # GET SUBTITLE
 # GET DATA
...

Title

Firstly, it would make sense to get the title of each article. Scanning the XML document, we can observe that all titles are found in a <title> tag

<item>
<title><![CDATA[ The Powers of Two: Why Is 1 + 2 + 4 + 8 + … = -1 ]]></title>
...

To retrieve this, all we have to do is use BeautifulSoup’s ‘find’ attribute (there’s only one title per article, so no need to use find_all)

title = a.find('title').text

U+1F4A1 The .text just removes the tags, in this case <title> and </title>, leaving us only with the text that we need

Subtitle

<item>
...
<description>
<![CDATA[ <div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/cantors-paradise/the-powers-of-two-why-is-1-2-4-8-1-19d8f00be228?source=rss-46b79b6c143b------2"><img src="https://cdn-images-1.medium.com/max/2048/1*geeybYiAeCd1vFkKuO75lA@2x.jpeg" width="2048"></a></p><p class="medium-feed-snippet">On calculating infinite divergent series sums</p><p class="medium-feed-link"><a href="https://medium.com/cantors-paradise/the-powers-of-two-why-is-1-2-4-8-1-19d8f00be228?source=rss-46b79b6c143b------2">Continue reading on Cantor’s Paradise »</a></p></div> ]]>
</description>
...

It’s good practice to put a subheading on our Medium articles, so readers know what to expect from an article. Looking back at our RSS feed, we can find our subtitle in a paragraph tag. But there are multiple paragraph tags in the page, so we need a way to distinguish it from the others. This can be accomplished with the unique class=“medium-feed-snippet” – which we can see is assigned only to the subtitle – and the attrs parameter.

for subtitle in a.find_all('p', attrs={'class':'medium-feed-snippet'}):
 subtitle = subtitle.text

We used find_all here because using find leads to an attribute error:

subtitle = a.find('subtitle').text
 AttributeError: 'NoneType' object has no attribute 'text'

Feature Image

<item>
...
<description>
...
<img src="https://cdn-images-1.medium.com/max/2048/1*geeybYiAeCd1vFkKuO75lA@2x.jpeg" width="2048">
...

The feed also provides us with the image source. We can just apply what we’ve done so far, except we use a [‘src’] key to get only the link to the image:

for img in a.find_all('img', src=True):
 img = img['src']

Link To The Actual Medium Article

<item>
...
<link>https://medium.com/cantors-paradise/the-powers-of-two-why-is-1-2-4-8-1-19d8f00be228?source=rss-46b79b6c143b------2</link>
...

To keep things short, we won’t be scraping the entire article content, but we can get the link to the medium article. This is helpful later on when we list these articles on a separate website (say, a blog or portfolio), where we can redirect viewers to the full article. Again, it’s almost the exact same thing as with the feature image, except we use [‘href’] key instead.

for link in a.find_all('a', href=True):
 link = link['href']

Publisher

Finding the publication where the story was published can be a bit more tricky. The publication name is not directly specified. But with a bit of ingenuity, we can see that the publication can be found through the article link. Hence, given the publications, a user submits to, and the respective publication slug (the ‘towards-artificial-intelligence’ part in medium.com/towards-artificial-intelligence), we can run through the possible publications as follows:

if 'towards-artificial-intelligence' in link:
 publisher = "Towards AI"
 elif 'cantors-paradise' in link:
 publisher = "Cantor's Paradise"
 elif 'mindreform' in link:
 publisher = 'MindReform'

Tags

<category>
<![CDATA[ science ]]>
</category>
<category>
<![CDATA[ infinity ]]>
</category>
<category>
<![CDATA[ math ]]>
</category>
<category>
<![CDATA[ education ]]>
</category>
<category>
<![CDATA[ numbers ]]>
</category>

Scraping the Medium tags can help is incorporate a search function in our website later on. Upon scanning the document, we see that the tags for an article are each nested in a <category> tag. Since an article can have multiple tags, we’ll use find_all and append each tag to a list like so

tags = []
 for tag in a.find_all('category'):
 tag = tag.text
 tags.append(tag)

Date

<item>
...
<pubDate>Wed, 01 Jul 2020 08:36:11 GMT</pubDate>
...

And last but not the least, the date. We have to be careful when parsing the date. Not only should we take note of the date format in the document, but also the format we want to convert it to (especially when dealing with databases). We will use the datatime module to do this.

import datetime # DON'T FORGET!date = a.find('pubdate').text
 date = str(date).replace(' GMT','') # REMOVE GMT STRING
 date = datetime.strptime(date, '%a, %d %b %Y %H:%M:%S') # CONVERT date TO DATETIME OBJECT
 date = date.strftime('%Y-%m-%d') # CONVERT TO DJANGO DateField FORMAT

As we can see from within the <pubdate> tag, the date is in the format “weekday, day month year hours:minutes:seconds GMT”. GMT can be disregarded, so we’ll remove that entirely. With this, let’s convert the remaining data into code strptime() can understand (check this link for full date format code).

'%a, %d %b %Y %H:%M:%S'

U+1F4A1 Note that decimal numbers in the RSS feed are zero-padded (e.g. 01 instead of just 1)

Finally, let’s convert it into our desired format, I simply followed Django’s default DateField format, which is “year-month-day” or

'%Y-%m-%d'

Organizing Data

That’s all!

Now we have scrumptious data on each of our Medium articles. Unfortunately, the feed does not give all our articles, but this shouldn’t be a problem in the future since we will be scraping this regularly.

To keep our output clean, let’s pass all of this data into a dictionary for each article, and in turn, pass that into a list:

 ... # EVERYTHING WE DID UP THERE
 # PUT ALL INFO IN A DICTIONARY
 article_info = {
 'title':title,
 'img':img,
 'date':date,
 'subtitle':subtitle,
 'publisher':publisher,
 'link':link,
 'tags':tags 
 }
 # PUT article_info IN article_all
 articles_all.append(article_info)
print(articles_all)

The Final Code

Here’s all our code put together. The final result is a list of all our articles — each represented by a dictionary containing the title, subtitle, image, link, date, publisher, and tags.

import requests
from bs4 import BeautifulSoup
import pandas
from datetime import datetime# GET ACTUAL CONTENT
url='https://medium.com/feed/@joaquindecastro'
response = requests.get(url)
content = BeautifulSoup(response.content, 'html.parser')# REMOVE UNNECESSARY STRINGS
content = str(content).replace('<![CDATA[','').replace(']]>','')
content = BeautifulSoup(content, 'html.parser')# GET LIST OF ALL ARTICLES 
# FOUND IN <item> TAG
articles = content.find_all('item')# CREATE ARRAY FOR ALL article_info
articles_all = []# GET ARTICLE INFO PER ARTICLE
for a in articles:
 # GET TITLE
 title = str(a.find('title').text)
 # GET SUBTITLE
 for subtitle in a.find_all('p', attrs={'class':'medium-feed-snippet'}):
 subtitle = subtitle.text
 # GET IMG SOURCE
 for img in a.find_all('img', src=True):
 img = img['src']
 # GET DATE
 date = a.find('pubdate').text
 date = str(date).replace(' GMT','') # REMOVE GMT STRING
 date = datetime.strptime(date, '%a, %d %b %Y %H:%M:%S') # CONVERT date TO DATETIME OBJECT
 date = date.strftime('%Y-%m-%d') # CONVERT TO DJANGO DateField FORMAT
 # GET LINK TO MEDIUM ARTICLE
 for href in a.find_all('a', href=True):
 link = href['href']
 # GET PUBLISHER BASED ON LINK (eg medium.com/my-publication/article-slug)
 if 'towards-artificial-intelligence' in link:
 publisher = "Towards AI"
 elif 'cantors-paradise' in link:
 publisher = "Cantor's Paradise"
 elif 'mindreform' in link:
 publisher = 'MindReform'
 # GET LIST OF TAGS
 tags = []
 for tag in a.find_all('category'):
 tag = tag.text
 tags.append(tag)
 # PUT ALL INFO IN A DICTIONARY
 article_info = {
 'title':title,
 'img':img,
 'date':date,
 'subtitle':subtitle,
 'publisher':publisher,
 'link':link,
 'tags':tags 
 }
 # PUT article_info IN article_all
 articles_all.append(article_info)
print(articles_all)

Here’s the Github repository: https://github.com/JoaquindeCastro/medium_scraper

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI