On behalf of Dan Bleaken, Senior Malware Data Analyst, Symantec Hosted Services
Recently it has been widely reported that global spam volumes have decreased, especially on Sunday 3rd October 2010, when spam levels dropped to their lowest for some time. This week spam volumes seem to be creeping back to normal levels. At Symantec Hosted Services we have a wealth of data on spam traffic, and crucially what contribution to global spam each of the major botnets makes. This blog will take a close look at botnet spam, what factors influence botnet output, and will try to explain some of the changes that occurred around the 3rd October.
The big picture
Nobody can be certain what the true volume of spam in circulation is, but some organisations, like Symantec, have a particularly high exposure to global email traffic. Symantec analyse email traffic that is seen in their global infrastructure and apply reasonable extrapolation to estimate global email and spam volumes. Estimates of global spam and email volumes from other organisations have to use a similar approach as no single organisation sees all email traffic. Some base the estimates purely on relative changes, having decided some time ago what the benchmark global spam and email volumes should be. All approaches have their merits. This is Symantec’s view of spam and email volumes both long-term (last 8 years), and during the last 12 months.
Monthly volumes (billions), Jan 2002-date.
Daily Volumes (billions), Oct 2009-date.
Figure 1. Symantec’s estimates of total email and spam volumes Jan 2002-date.
The trends in Figure 1 show the meteoric rise of spam to the levels experienced during the last few years. Typically in 2010 we estimate that between 100 and 200 billion spam emails are sent globally each day. It shows the huge (approximately 80%) drop due to the takedown of Californian ISP McColo in November 2008, which crippled several of the major botnets. It also shows the wandering nature of global spam volumes as botnets (responsible for 80-90% of all spam) battle for dominance, evolve, or suffer various setbacks.
Botnets drive global spam volumes
Botnets are enormous networks of infected PCs, and under the control of criminal gangs they can be instructed to perform certain tasks on a massive scale. They hide the identity of the attacker, because the machines are owned by innocent users all over the world, they could be in your office, in grandma’s back bedroom, anywhere. And these are people who have at some point been the victim of malware and had their PC turned into a bot. Their distribution is like a rash on the world map: where there are PCs, there are bots.
Figure 2. The source of botnet spam, August 2010
Botnets are a major element of the shadow economy infrastructure, and a powerful tool, mostly used to send millions of spam/malware/phish emails, to insert malicious code into websites, and perform Distributed Denial of Service attacks (DDoS) – to attack websites, servers, and even entire countries.
Botnets send such a vast volume of spam each day that other sources of spam fade into virtual insignificance. If there is a major shift in spam volumes, it is almost certainly due to a change in output of one or more of the major botnets. There are perhaps 10 or 12 botnets that lead the way, with 2 or 3 mega-botnets that out spam all the others.
Figure 3. % of spam from each botnet, July 2008-date, moving 30-day average. Currently (in order), Rustock, Grum, Cutwail, Maazben and Mega-D send the most spam.
Figure 4. Key botnet stats, spam sent and estimated size measured in June 2010.
The above figures show that the major spamming botnets have a variety of sizes and outputs, and it changes. For example (see Figure 4), in June Rustock was responsible for 41% of global spam, which we estimate equates to 46 billion spam messages sent daily, 32 million spam per minute. Considering how much spam each individual Rustock bot (infected PC) sends, we estimate 192 spam per minute. Imagine your PC pumping out 192 spam per minute, to unfortunate recipients all over the world. In June Rustock achieved this with an estimated 1-1.5 million bots. Other botnets send less, but instruct individual bots to work harder than Rustock, for example, Mega-D. Others have a huge amount of bots at their disposal but pump out spam more steadily, for example Lethic. This situation changes over time. Some of the factors that affect the size and output of botnets include:
- Regional bot infections – number of active bots is influenced by:
- Level of internet user awareness (low awareness = more likely to become infected by malware and recruited to a botnet);
- Level of protection applied by users against internet threats (for example anti-spam, anti-virus, protection from web, instant messaging and endpoint threats);
- Level of technological development of regions (for example in regions where broadband access is being rolled out rapidly, large number of new users are getting online);
- Population of the region – highly populated regions generally have more internet users and more bots. But there are exceptions; in some cases highly populated regions have poor broadband penetration.
- Botnets have various issues in keeping their infrastructure ticking along nicely:
- issues relating to action taken against them or important parts of their infrastructure;
- Errors – botnets do mess up! We frequently see broken spam runs;
- Who is awake? PCs (bots) have to be turned on to send spam – throughout the day most spam emanates from regions in waking hours, especially working hours: more PCs are switched on. Of course, many PCs are left switched on all the time. At weekends spam volumes almost always decrease with more bots switched off than during the week (Mon-Fri).
- Business & sophistication:
- the demand from spammers (botnet’s customers) changes over time;
- also demand from customers requiring other uses for the botnet;
- the level of sophistication of the botnet, including how it protects itself from disruption from law enforcement activities;
- How aggressively the group behind the botnet recruit new bots (infect PCs) versus bots lost (PCs cleaned/malware removed).
- The botnet’s choice:
- Ultimately the groups behind botnets have the choice to allocate their resources to different tasks, and so the proportion of each botnet active in spamming will shift.
- Spamming
- Distributed Denial of Service (DDoS) attacks
- Theft of personal information
- Mass infecting websites to serve malware
- Other
- Botnets also have the choice to throttle their output to lay low or to go all-out to send spam. They might be influenced by heightened awareness of botnets and/or law enforcement activity, or pump up the output in response to huge demand from queuing spam gangs.
- Issue affecting spammers:
-
- The groups running the botnet may set up and run their own spam campaigns, or ‘rent out’ part of their botnets to spam gangs who want to distribute their spam.
- If some commodity that the spammers rely upon suddenly becomes unavailable, the spammers can’t construct their spam campaigns, and botnets will experience a sudden dip in traffic as a result. Examples of spammer commodities are email distribution lists and affiliate programmes that provide the focus of the messages sent by spammers.
What have we seen recently, especially around the 3rd October?
Essentially only a few of the most dominant botnets are capable of influencing global spam volumes, the smaller botnets may produce a measureable change but not of a magnitude that would register with many people. Last weekend’s spam decrease (3rd October) was picked up clearly by many security vendors and by other interested parties running spam honeypots and so on.
Aside from the question of how much spam is circulating globally, we can gain some valuable insights into the factors influencing changes in spam volumes by looking at relative changes rather than absolute values.
We can look at the relative dominance of botnets in terms of the percentage of global spam that they send, which is interesting but has its limitations, as I’ll describe below. We can look at the relative output of each botnet in terms of ‘spam per minute’, in other words how much each botnet is sending irrespective of other botnets. And finally for each botnet we can examine ‘spam per bot per minute’*, which gives us a rough feel of how hard individual bots are spamming in each botnet.
Figure 5. % of spam from each botnet, August 2010-date
Figure 5. is the most recent segment of Figure 3, and shows the % of global spam sent by each of the major botnets. We can see some interesting features.
Rustock, having enjoyed a rapid rise to dominance during the last 2 years, was by far the dominant botnet in August. However in September, Rustock appeared to hit some turbulence.
We can see Cutwail has fairly consistently represented about 5-10% of spam, and can clearly see the impact of the attempted takedown of Cutwail at the end of August. Following that, it didn’t take long (a few days) for Cutwail to return to business as usual.
There are all sorts of changes going on, but the focus of this blog is what happened last weekend and on the 3rd October.
Before continuing, it’s important to note that despite giving us an interesting view of the relative dominance of the major botnets, this view has its downsides, as if the % of spam from a given botnet changes, that could be because the spam output of the botnet has changed, or it could be because the output of one or more of the other botnets has changed. So it’s useful to also consider the actual output of each botnet, in terms of ‘spam per minute’.
Figure 6. spam per minute from each botnet, August 2010-date
Looking at each botnet’s output in terms of spam per minute, the first thing that is striking is that since mid-August there has been an overall gradual decline in spam volumes.
No single botnet is responsible, but we can see that Rustock’s output has decreased, as well as the output of Mega-D and Grum. At the point Rustock began its decline at the start of September, we registered a large volume of spam as ‘unknown/other’. We believe that in this data some of the changes we saw in Rustock output in Figure 6. were attributable to the Rustock botnet using some new message variants (which for a short period we would register as ‘unknown/other’), however on several occasions we saw Rustock drop all output for long periods (e.g. 12 hours), which could mean one of many things, such as issues with the botnet, the botnet being used for some other purpose, perhaps even botnet-wide updates to improve its efficiency. Ultimately, whatever botnet we attribute the spam to, the combined output of the major botnets declined.
We can see that on the 3rd of October, when global spam volumes hit their lowest point for some time, that decreases in the output of two botnets were largely responsible.
- spam classified as ‘unknown/other’ dipped strongly (most of which we believe to be Rustock), as well as the output that we label as ‘Rustock’;
- spam classified as Cutwail dipped strongly.
Other than Rustock and Cutwail, Grum was very active at the time, but didn’t show a dip on the 3rd October. Grum showed a small dip the next day, but quickly went on with its normal output soon after.
So the output of Rustock and Cutwail decreased on Sunday 3rd October. But was this due to fewer bots sending spam, suggesting a portion of the botnets had been re-assigned to some other task, or lost? Or was it due to individual bots sending less (but the normal number of bots sending)? To answer this we can look at the ‘spam per bot per minute’.
Figure 7. spam per bot per minute from each botnet, August 2010-date
Figure 7. looks much ‘calmer’ than Figures 5 & 6. Generally we don’t see the amount of spam per bot per minute change much for most botnets. There are the occasional blips, where botnets clearly instruct bots to spam harder, or spam less hard. But generally bots work at a similar rate for months on end.
In terms of Rustock and Cutwail, we didn’t see any particular shift in the number of spam sent per bot per minute on the 3rd October. Ok, maybe a very small decrease for Cutwail. We think that fewer bots were sending spam for both Rustock and Cutwail on that day, causing the decrease in output. This would happen if bots were reassigned to some other task, the bots were lost, or if demand from spammers simply wasn’t there.
The hourly detail
Let’s dig deeper into the major botnet’s output in and around last weekend. As discussed above we think that the drop in global spam volumes was due to a decrease in output by Rustock, and Cutwail. We also think that fewer bots were sending spam, rather than the normal number of bots sending less.
Figure 8. hourly botnet output during the last 7 days
One part of Figure 8. stands out: Sunday 3rd October. We can see that the output of Rustock went to zero at about 8am GMT. In fact we make it 8:47:09 GMT. Rustock started to send spam again at 23:02:46 GMT. For Cutwail, it’s harder to see the drop but it is there. It’s easier to see it on the figure below.
Rustock
Cutwail
Figure 9. hourly botnet output during the last 7 days for Rustock, and Cutwail
Figure 9. shows Rustock’s break again, and it is now easier to see Cutwail’s reduction in output, albeit small.
Why?
So why did Rustock and Cutwail reduce their output on Sunday 3rd October, driving a large drop in global spam volumes?
Well many factors influence how much spam a botnet sends, some of those factors are discussed earlier in the blog, such as takedown attempts, shifting use of the botnet, changing demands from spammers and so on.
Something that has arisen in the news this week that could be important is the closure of a notorious spam affiliate called ‘Spamit’ (
http://www.theregister.co.uk/2010/10/06/spamit_shuts_up_shop/). Spamit was the mainstay of the so-called ‘Canadian Pharmacy’ business. Approximately two thirds of all spam is related to pharmaceutical products, and a great deal of that is related to Canadian Pharmacy websites and related brands, which sell a variety of pills/drugs for anything from male enhancement, to weight loss, to stress relief. It’s an enormous money making machine in the shadow economy, and spammers line up to work with affiliate schemes such as Spamit, distributing enormous volumes of rapidly changing spam and taking commission for their efforts.
Figure 10. spam category break-down
The loss of such an important commodity for spammers would certainly have an impact on global spam volumes. Spammers with campaigns designed and queued for distribution would rapidly bail out and look for alternative ways to make their money, possibly leaving botnets a little in the lurch. Botnets orchestrating their own spam campaigns may equally halt their activities while they consider their next move.
If it was the loss of this affiliate that caused the drop in output of Rustock and Cutwail, then we have evidence to suggest that Rustock is very likely to have experienced more disruption than Cutwail, which it certainly seemed to.
Let’s have a look at which botnets send the most pharmaceutical spam.
Figure 11. % of pharmaceutical spam from each botnet
So Rustock and Cutwail clearly have a major contribution not only to spam, but to pharmaceutical spam. But how much of the Rustock and Cutwail spamming operation is related to pharmaceutical spam?
Figure 12. Category of spam sent from Cutwail and Rustock
Ouch. If the spammers using Rustock were heavily reliant on the affiliate Spamit, they would have had quite a shock when Spamit was closed down. Cutwail on the other hand, has its ‘fingers in many pies’ spam wise, sending pharma spam, watch spam, phishing and malware among other things. Cutwail spammers would have also had a surprise, but many Cutwail spammers would have happily carried on with business as usual. Perhaps the heavy reliance of Rustock on pharma spammers, and in turn their reliance on Spamit, was responsible for Rustock’s rapid shutdown on Sunday, as they scrambled to roll back planned spam campaigns.
Symantec Hosted Services protect their clients from a blizzard of botnet spam every day. Sophisticated botnet intelligence is applied to spot the characteristics of botnet spam instantly, and eliminate it comfortably before it reaches the client’s network.