The hidden costs of AI on the web

Have you noticed how you may see more server errors instead of the content you tried to view on some sites?

Or - if you run your own WordPress blog - that your WP admin is much more likely to error out when doing tasks that previously just worked?

Welcome to the new and exciting world of AI biting the hands that feed it!

I have been seeing this a lot recently, but it's greatest harm isn't to the bigger sites: No, the most harmed corners of the interwebs are the tiny little niche sites running on shared hosting plans.

Here's what is happening:

In the past, website crawlers were just your basic search engines or archival bots. They showed respect to the sites they visited by limiting how often they visited, how fast they crawled the pages, and in general were few enough in number that you'd never notice them whether they were filtered out of your analytics reports or not.

Now, however, we've got corporations as well as state-level actors running bot farms that are absolutely hammering the entire internet to consume as much as they can.. And then they crawl it again, and again, and again, just to see if anything changes.

For statically generated sites like mine (speaking of, did you notice the redesign? :D), this isn't too much of an issue.. And Nerfed Gamer News? It does pretty well, but I also have it heavily optimized and am running a flat file CMS (GravCMS) that is very performant in general.

Some of my sites running a custom CMS I built, however? Yeah, I occasionally get downtime reports due to the heavy load they get from AI bots (thankfully not nearly as much as a WordPress site, however).

Sadly, for the average person, there isn't a lot that can be done; While you could sign up for a free CloudFlare account and go through the headache of setting that up in order to block AI crawlers, CloudFlare hasn't been without it's own issues lately, too.. And the average person is not going to be able to understand how to configure even a basic CF setup.

Even when you have this setup, though, there are bots that still get through.

AI is effectively DDoSing the niche internet, and they don't care..

..After all, you can always ask them for the content, right?

Sure you can.. until the niche network of novelties on the 'net are no longer there to be crawled.