Many people are genuinely shocked when they are told the fact that >50% of website traffic is actually non-human. In 2016 this number was 52% according to a survey by Incapsula.

“More than Meets the Eye”

Don’t be alarmed, it’s not an army of super intelligent sewer rats which have gnawed their way into the cables which zip the bits and bytes around our towns and cities – browsing for rodent related material. Neither is it extraterrestrial. This non-human traffic is comprised of bots. But what are bots exactly.

Bots are small computer programs which run automatically to serve a specific purpose. Largely bots can be divided into good-bots and bad bots. Think about Transformers; the good guys are the “Autobots”, the bad the “Decepticons”.

The good bots are things like spiders or search engine bots which index and classify the web. These are as essential to the web as a librarian is to a library. (Think of having to find a book in a library without a system to help you!) Other “good” bots would include fetcher bots and general monitoring bots. These collectively serve a value adding purpose to a well functioning internet.

Then there are the bad bots. Trojans, DDoS, Password hackers. The “Decepticons”. These have nefarious intent and can do some serious damage. Remember the massive outage on October 21 2016, which used a botnet of IoT devices to take down large parts of the internet in the US?


Robots in Disguise: The “Autobots” vs the “Decepticons”

Map of Areas most affected by the attack on 21 October 2016

The bad news… a whopping 56% of the bot traffic is estimated to be bad bot traffic. The Association of National Advertisers and White Ops have estimated in a study that ‘Ad Fraud’ bad bots have had a significant cost impact on advertisers. The study estimates that more than $7 billion annually is lost through bot based advertising fraud.

Can you protect yourself? In a way, yes. But I’ve heard this exercise being compared to a game of Whack-a-mole! As soon as you take out one IP address, several others will pop up in their place. Be vigilant and put your best efforts in to reduce the impact of this menace. Coincidentally, it is the smaller websites which may suffer more from bad bot traffic, as they do not allocate the resources to vigilism in this regard. MOZ has a great blog post which goes into a little more detail on prevention here. Then there are the vendors; IncapsulaCloudflare. Also CopyScape has a useful service to identify which plagiaristic sites that have stolen your content.

Roll Out: but be wary of the decepticons.