Three Ways to Fight AI Crawlers

Author(s): Lo Zarantonello

Originally published on Towards AI.

Why AI crawlers are the free-riders of the web and how to deal with them

Until recently, websites were pushing for web crawlers to index their content properly.

Now, a new type of crawler, AI crawlers, is changing the game, with negative repercussions on open source content and increasingly on companies that rely on content.

Top AI Crawlers by Request Volume — DigWatch, Cloudflare

AI crawlers are increasingly becoming a significant challenge for website owners, consuming resources and scraping content without permission or compensation.

In a recent TechCrunch article, you can see how AI crawlers are causing issues, especially among open source developers.

AI bots don’t honor the Robots Exclusion Protocol robot.txt file, the file that tells bots what not to crawl, and it shows. People report:

Spikes in costs for the website ownersOutages or performance issues for the usersDDoS outages — In the worst cases

Want a tangible example?

Look no further than TechPays.com!

TechPays.com

The founder of the website noticed an over 10x increase in data outbound and over 90% of the traffic was AI crawlers.

Last month AI crawlers generated 90% of my site’s traffic

AI crawlers like Meta AI, ImagesiftBot, DotBot.

According to Cloudflare, AI crawlers generate more than 50 billion requests to the Cloudflare network daily — almost 1% of all web requests!

Because the content is scrapped for free and will then be… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI