July 17th, 2009 at 04:44 UTC by Richard Clayton
On 11 November 2008 McColo, a Californian server hosting company, was disconnected from the Internet. This took the controllers for 6 major botnets offline. It has been widely reported that email spam volumes were markedly reduced for some time thereafter. But did disconnecting McColo only get rid of “easy to block” spam?
In a paper presented this week at the Sixth Conference on Email and Antispam (CEAS) I examined email traffic data for for the incoming email to a UK ISP to see what effect the disconnection had.
The first thing to note is the amount of spam — which was high (and highly variable from day to day) — reduced sharply. The figure shows the amount of spam each day in weeks 1 to 7. McColo was disconnected late on the Tuesday of week 6.
However, several heuristics cause spam to be rejected before an (expensive to run) content filtering stage: email from sites in the SpamHaus Policy Blocklist (PBL) is discarded; greylisting is used; and the ISP is picky about SMTP protocol compliance. Before McColo’s disconnection, between 32% and 56% of the spam did not need to be assessed by the content filters; thereafter a more consistent 43% or so was detected before the content filter stage. The second figure shows this, the key message being that although spam was less, a lot of the huge variation from day to day was email that didn’t need to be assessed by the content filters.
The ISP applies some special rules for subsets of customers. The next graph (note the different vertical axis) show how much email could be discarded because it was sent to non-existent addresses. On some days this mechanism immediately discards 900,000 emails. However, once McColo has been shut down, a mere 50\,000 or so emails a day are blocked.
Spam to a few customers is rejected if the sender IP address is on a blacklist. This mechanism was being tweaked during the study period, so data is only valid during the first half of November. Nevertheless, the last figure shows a clear effect when McColo is shut down. The decrease is proportionately more than the overall drop in spam, meaning that blacklists ceased to be as useful in the immediate aftermath of the McColo closure.
The disconnection of McColo was obviously a Good Thing, because of the substantial, albeit temporary, reduction in spam. However, particular types of detection mechanism ceased to be as effective. Headlines of 60+% spam reduction only tell one part of a complex story.
Entry filed under: Academic papers