Dōmo arigatō

I found this interesting and wanted to point out that your sites are likely receiving a butt-ton of traffic that you just don’t see. Because bots, scrapers and security hack attempts don’t use a browser in the same way that human’s use a browser, they rarely show up in your stats. And when they are legit bots (search, feed finders, approved AI looking for answers), that’s how it should work.

It’s when they are not legit bots and malicious actors that it becomes a problem. I don’t know how wordpress.com hosted sites handle it, but I am with a different hosting service and decided I was tired of scrapers and illegitimate bots trying to access my site.

So my site content is delivered via a proxy you may or may not have heard of: Cloudflare.

I’ve set up some security rules since sending traffic through Cloudflare’s proxy services to hopefully limit the bad actors that show up. It has helped limit or eliminate non-human visitors that were previously accessing the site. While this snapshot is not as “clean” as I would like (I modified the rules several times over the last 30 days), it nevertheless gives you an idea of what is going on in the background that you can’t see when you host a WordPress site.

First off, there’s the big number. Almost 300,000 requests for site pages or content. I promise you… I have never seen anything close to that kind of traffic in my human-focused stats. The best I saw when I was getting a lot of “human” traffic was ~6,000 hits/month.

This is a snip of the security panel, so just to the right are the number of mitigated requests — those requests that were served some kind of security-oriented product instead of the content. I am confident that near-zero humans were blocked because of the security definitions (except when they were impatient for the bot check to be completed).

Doing the maths on it, that is roughly 6% of all requests that were likely bad actors coming to the site.

I expect that percentage to increase in the next 30-day window.

Most of it was data-center scrapers or bots trying to find vulnerabilities in plugins or code. Cloudflare keeps track of that as well, so I kind of know the source.

For someone like me, who likes to interpret data and information, it was an eye-opener. I had no clue just how many bots were visiting the site: how many were legit and how many malignant. It’s far more than I ever guessed of either type.

Now, if only humans found the site quite so interesting.


3 responses to “Dōmo arigatō”

  1. Chico’s Mom Avatar

    I’ve noticed that my site is getting a crap ton of hits.

    1. michael raven Avatar

      I’m betting a lot of those mystery hits are from datacenters like Tencent or even Google’s rented server spaces. My stats took an immediate hit, but they were probably not very real. Phishing and scams looking for emails to scrape when you go check out their site and comment.

      1. Chico’s Mom Avatar

        My real readers know who they are and I thank them. 🏆

Leave a comment. Markdown permitted.

This site uses Akismet to reduce spam. Learn how your comment data is processed.