The Prevalence of Threats: How Popular is the Malware That Infected Your System?

In my last blog post, I talked about the infection rates of Immunet users (namely looking at how many threats we blocked on different machines).  In this post, I’d like to dive into how frequently a piece of malware might show up on a given system. 

I took a look at data queried to Immunet’s cloud from between September 15, 2010 and October 15, 2010 (i.e., a one-month period), and looked specifically at convictions issued through our basic 1-1 signatures.  As of October 15^th, we had approximately 580,000 users and were tracking about 16 million unique threats.  (As of today we have over 620,000 users and data on around 16.5 million threats.)  

For the purpose of this exercise, I was more interested in seeing how prevalent known threats were, which is why I only examined results from our basic 1-1 signature approach.  (Note that detection against known threats is only a part of Immunet’s overall protection stack; for example, we have technologies like Ethos and Spero that are designed to catch threats that were not previously known to us.  Check out my other blog post on Ethos and Spero).  The results are graphed below.

Of all the threats we identified (by SHA-2), slightly less than half were on more than one system.   The remaining threats were seen exactly once and never again.   At the other end of the spectrum, less than 1% of the distinct threats we saw were on more than 23 machines.  Note that the results are cumulative, so the bars will add up to more than 100%.  (For example, a threat that is on three machines will be counted in columns one and two.)

These findings have a few implications.  First, they point to a significant shift in balance of the economics of traditional push-based signatures that many incumbents in the AntiVirus space use.  Does it really make sense to push out tens of thousands of new AntiVirus definitions to millions of users (as many vendors are doing) when 99% of these threats will trigger on less than a couple of dozen machines?  From the data we are seeing, the answer is an overwhelming no.  In this capacity having a cloud-based pull approach provides far more favorable economic tradeoffs. 

Second, these findings point to the highly transient nature of threats (a topic that I will definitely cover in a future blog post).    Many existing companies in the AntiVirus space suffer from significant latencies in their back office operations.  From the time that they know about a threat internally to the time users are protected against these threats, anywhere between tens of hours to tens of days could pass.  Given how many threats are detected on just one machine, this approach is a classic case of “too little too late.”  At Immunet we’ve built an architecture that inherently eliminates these inefficiencies, and yields a near real-time feedback loop.

Finally, these findings cement the argument that a pure signature-based approach has limited value.  Signatures work especially well for popular threats.  They are conceptually easy, well tested, and are targeted enough to ensure a low rate of false positives (i.e., a case where a legitimate application was mistakenly called bad).  However, they fail miserably for fly-by-night threats.  This shortcoming of signatures is one reason why we developed approaches like Ethos and Spero (among others) for finding ephemeral threats. 

Ultimately, it’s obvious that the old way of doing things is not working, and a changing of the guard is necessary.  Although the data we are seeing makes it abundantly clear, the reality is that the writing has been on the wall for far longer.  We formed Immunet to address shortcomings that we’ve seen in the industry for some time.  As we catch threats that others miss, we can score one more for the good guys.   There is an amazing feeling in knowing that we might have just protected someone from getting their identity stolen or having their computer usurped for more nefarious purposes.     

Is Your PC Haunted?

The fact is your PC could be haunted by all manner of malicious code from zombies, botnets and worms to viruses, Trojans, spyware and keyloggers—and not just at Halloween. Anyone who’s ever been attacked by any of these vile creatures knows it’s downright frightful. The thought of your PC becoming infected and then infecting those of your family, friends, neighbors and acquaintances is enough to make you scream! Even more horrifying is losing your hard drive and everything on it: family photos, files, your address book—a shocking experience that is a very real possibility without the proper antiviral safeguards.

Then there’s the criminal aspect: Trojans, which are spread by email attachments and bogus Internet downloads, turn PCs into zombie machines, enabling their creators to suck the good stuff out of your computer and pass it on to the bad guys for all manner of mischief. A botnet is a group of connected zombie computers that can be used for spamming and mass credit card fraud, among other things. It can take less than five minutes of time online for a Trojan virus to take hold of an unprotected computer and use it for illicit activities—and the owners rarely suspect it. According Microsoft’s just released Security Intelligence Report over 2.2 million PCs in the U.S. are infected by botnet viruses.

While there are no silver bullets to protect your PC, there is Immunet. Think of it as the magic potion to shield your PC from malevolent cyber tricksters looking to wreak havoc on unsuspecting home computers – and not just yours, but everyone you communicate with via email or over the Internet. Social networking venues like Twitter and Facebook, and online games, are especially fertile breeding grounds for these creepy invaders when people don’t protect their PCs with AntiVirus software.

And here’s a scary thought sure to make you quake in your boots:  According to a recent USA Today article, an estimated 40% to 60% of PCs are completely unprotected. These machines are easy targets for cybercriminals. Infecting them, stealing data and using them to conduct online scams is as easy as sneaking chocolate out of your kid’s Halloween stash. If you’re socializing online with someone who has an unprotected PC that becomes infected, you and all of your friends are likely to be the next victims.

To counter the threat, Immunet developed a new way to keep personal PCs safe. Immunet’s Collective Immunity™ technology uses the power of the Internet (the “Cloud”) to create a network protection effect that keeps your PC and the computers of your friends and family safe online. Each time a virus is blocked on one computer in the Immunet Community ALL other computers are instantly protected from the same virus, increasing the speed and level of virus protection with each new member. This collaborative approach uses “strength in numbers” that grows smarter with each new community member.

Just like you’d never let your child go Trick-or-Treating alone, there’s safety in numbers when it comes to keeping your PC safe. Immunet Protect Free is today’s most innovative cloud-based anti-viral software for protecting your PC and social network against all manner of cyber threats from around the world. Within seconds of downloading it, you gain the protection produced by every other PC in the Immunet Community around the world—now nearly 600,000 strong, including Vatican City.

Unlike traditional antivirus products, which reside on PC hard drives, and take between one and 14 days to gather, analyze and deliver updates—often after the threat has mutated, Immunet identifies viruses in real-time and continually delivers antivirus protection via an Internet connection to its community of Immunet Protect users. Immunet Protect is up to 35 times lighter than traditional antivirus solutions, which take up between 100 and 350 megabytes of disk space and causes performance slowdowns. In contrast, Immunet Protect uses only 10 megabytes of space, to keep your PC running at its optimal speed.

And in the spirit of Cyber Security Awareness Month, we’re offering a 25% discount for anyone who wants to upgrade to Immunet Protect Plus in October. Just enter the promotional code “OCTOBER” at the Immunet Store during check out.

Immunet Protect Plus provides everything that Immunet Protect Free provides, plus a host of extra goodies like enhanced virus removal, the ability to schedule virus scans, offline scanning that protects your PC even when you’re disconnected from the Internet, and more—all without slowing your PC to a sluggish crawl!

Have a Zombie Free Halloween! Be Safe!

A Look at User Infection Rates

One thing that especially excites me about being at Immunet is that we have taken a data driven approach from the onset.  That approach allows us to gain incredible visibility into threat landscape trends.  Over the next few posts, I thought I’d describe some of the data we’re seeing in the field.

In this particular post, I’d like to dive specifically into user infection rates.  This topic is particularly relevant since I’ve seen many claims about this topic, some of which are outlandish and others which are actually realistic (even though they ostensibly appear outlandish).    To get a handle on it, I went through Immunet’s data store, focusing specifically on the period between August 15, 2010 and October 15, 2010 (i.e., a two month window), to see how many users had N or more threats blocked.  The results are charted below.  



As of October 15, 2010, Immunet had 580,000 users.   First, a whopping 39.11% of Immunet’s user base during this period had at least one blocked infection.   That number is staggering.  On the extreme side of things, about 5% of our users had 20 or more infections.  This subset clearly comprises users who consistently engage in the kinds of behaviors that get them infected (e.g., failing to patch their systems and applications, continuously opening attachments, clicking on suspicious links, etc.).   It’s important to note that since we have new users constantly joining (and since these users may not have been around long enough to have encountered a threat), the results are skewed and the situation is even worse than what might appear.

What is quite alarming about these numbers is that they most likely represent users who, on average, are more security conscious.  After all, our users are running AntiVirus software.  Furthermore, because our software can run in companion mode (and because our agent is lightweight), many of our customers actually run Immunet Protect alongside existing AntiVirus software.  So, in terms of being security conscious, our users (on average) are quite possibly the cream of the cream of the crop.   

Now, consider that the number of people on the Internet as a whole will likely surpass 2 billion by the end of this year.  Based on our assessments, about half of users overall don’t run any AntiVirus software at all.  If you assume that of these, half will get infected (which is quite conservative based on the 39.11% number we gave above), then you are looking at 500,000,000 users whose machines will be actively infected.  With these kinds of numbers it’s clear that the bad guys will be dealing with a more substantial big data problem than even we have to handle!

Ethos and Spero: A Quick Peek into the Immunet Protection Stack

In my last post I began talking more about our protection technology stack. The focus of that post was our more traditional detection mechanism based on 1-1 signatures. While we have done a tremendous amount to push the envelope regarding what traditional signatures can do (for example, by using a cloud publishing model that facilitates real-time protection), the fact is that the traditional 1-1 signature-based approach will never be a comprehensive solution to the vast array of threats our customers face on a daily basis.

To address that issue, we have a number of additional engines that form a part of our overall protection capabilities.  I’d like to respectively talk very briefly about two of them today, namely the Ethos generic detection engine and the Spero Machine Learning engine.

Ethos Generic Detections

Ethos generic detections go one step further and try to “generalize” existing traditional fingerprints. The idea is that even if a virus author made a number of shallow changes to a specific piece of malicious software as a means to circumvent signature-based detection (which is remarkably common), we can still catch this threat via Ethos.  At the “heart” of Ethos are automated algorithmic techniques for creating signatures that “withstand” such superficial changes.   While many vendors employ some form of generic detection, what makes the Ethos system unique is the level of automation we have built in around signature generation. 

Incumbent AntiVirus companies rely on human analysts to come up with generic signatures.   Immunet’s technology is algorithmic.  The result is that our customers are protected in near-real time (which is critical considering that threats are highly ephemeral and are often alive for just a few hours).  As before, we are able to accomplish this type of automation because of our data driven approach, which allows us to determine automatically the threats that are suitable for generic signature creation and to algorithmically create those signatures.  Furthermore, we can monitor how this technology is performing in the field, what threats it is catching, and how malware variants are evolving.

Spero Machine Learning Technology

The other critical piece of Immunet's offering is our Spero technology, which leverages machine learning techniques to detect malicious software.   While the general idea of using machine learning techniques in an AntiVirus technology is not new and while a small number of AntiVirus vendors appear to have begun leveraging these techniques, there are several aspects of our approach that are worth noting. 

First, we use actual field data as the basis for training and evaluating our classifiers.  Our ability to do so is a consequence of our cloud-based architecture, which gives us extensive visibility into our real-world performance.  Furthermore, since we have taken a data driven approach from the onset, we are able to effectively train on exactly the kind of data our classifiers will encounter in the field.  (Without this type of architecture, machine learning techniques – as powerful as they are – would essentially amount to shooting in the dark.)  Having the right data source is the foundation on which machine learning techniques must run.  Without good data, trying to apply changes and tweaks further down the stack is a pointless exercise in futility (that far too many people seem to be addicted to). 

Second, we also leverage a number of distributed computing, advanced data mining, and machine learning techniques, such as Map Reduce, feature selection, and cost-sensitive classification.  In this regard, we have benefitted tremendously from the recent “big data” movement that has resulted in advances in the underlying algorithms, frameworks, and available tools for being able to process, analyze, and visualize large data sets.   

Overall, Ethos and Spero have been very powerful technologies for us.  They have given us a remarkable window into the vast spectrum of threats that evade traditional signature-based approaches. Time and again we have seen these technologies identify threats that many vendors miss.   Beyond these technologies, we do have a number of others that I can’t go into just yet.  But what I think makes Immunet so compelling is that our underlying architecture correctly looks at AntiVirus protection as a “big data” problem.  This architecture allows us to rapidly innovate and bring new ideas into the field.   In the race against the bad guys, I fundamentally believe that this approach is fueling our ability to break away from the pack.

The Immunet Protection Stack and Standard Signatures

In my last blog post, I described some of the overall design goals of Immunet Protect.  In the next couple of posts, I wanted to talk about some pieces of our protection stack in some detail.   It will not be feasible for me to describe all aspects of the technology (nor would it be advisable for me to talk about all the tricks we have up our sleeves!).

Our major protection technologies include, but are not limited to, traditional fingerprint-based protection, Ethos generic detections, and Spero machine learning based detections.  In this post I will talk about signatures and leave the other topics for the future.

Immunet's traditional fingerprint based detections are ostensibly similar to those seen in other AntiVirus engines.  They are geared towards specific threat instances, and essentially form an initial line of defense.  While signatures alone fail to provide comprehensive protection against the vast spectrum of today’s threats, they still are suitable for handling specific threat classes (such as widespread malware).  Furthermore, signatures are a well understood technology and tend to be less false positive prone than more generic approaches.  

There are two aspects of Immunet’s signature-based technology that are worth noting.  First, we have invested considerable effort into signature automation.  Consequently, our signatures are not hand built by analysts and as such the time it takes for us to make a signature available to our end customers is short.  In contrast, incumbent AntiVirus vendors have not streamlined this process and incur latencies that can be measured in the tens of hours to tens of days from the time they first learn about a threat to the time a customer is protected against it.  Second, we use a cloud-based delivery model, which allows us to make protection more readily available to our customers. 

These two facets are especially germane under today’s threat landscape where the vast majority of threats are fly-by-night and will likely be encountered on a small number of machines (if even encountered at all) for a brief period of time.  Because we have a cloud-based solution and a vast signature database, we have unique visibility into threat lifetimes.  Based on our actual in-field data, approximately three-quarters of threats that we observe in the field (and have signatures for) are seen just once – meaning the first time and the last time we see them are one and the same.  Of the remaining, approximately 10% have a lifetime of less than 10 hours – meaning that the last time one of our users encountered the threat was approximately 10 hours after the first user encountered it. 

Therefore, the process of translating “back-office” knowledge of a threat into a customer facing detection mechanism needs to be as swift as possible.  Furthermore, it turns out that only a small fraction of the threats that we have visibility into in our back office are ever seen in the wild on actual customer machines.  Among threats seen in the wild there appears to be a power law distribution where a small fraction appear on a large number of machines and a large fraction appear on a small number of machines. At the same time, it is not always possible to determine up front which of these threats will be runaway successes and which will more-or-less be one-hit wonders.

Consequently, the standard “push” model in which AntiVirus signature update packages are sent to each user is impractical since the likelihood that a given user will actually encounter a given threat is very small.  Instead, we employ a pull model via the cloud.  As I alluded to earlier, beyond being computationally efficient, this model also allows us to gather real-world data on how our technologies perform in the field.  This type of view into actual field operations is unparalleled among existing AntiVirus offerings.  While existing vendors have some data collection systems, these systems are often disparate to the point where the underlying protocols, programming languages, databases, schemas, data formats, operating systems, operating procedures, and operational personnel are different.  To make matters worse, personnel who might have initially developed some of these systems may be long gone.   Even still, these systems were not all developed with the purpose of creating a tight feedback loop between in-field data and actual protection mechanisms.  A data  driven approach must be a fundamental design principle from the onset – it cannot simply be thrown on after the fact.  Companies wishing to employ a model like this (who are not already doing so) will need to overhaul technical systems, deal with large-scale integration issues, and will likely need to significantly reorganize their operations.

Because we have taken a clean slate approach at Immunet, we are not encumbered by these types of legacy concerns.  Even though the industry as a whole recognizes the need to be data driven, we have been in a far better position to execute on that vision.   We replace the dozens of systems other vendors use with literally just a handful.  Furthermore, we established data models, schemas, and protocols that allow for seamless correlation between these systems.

Another benefit of our pull-based cloud model is that on the back-end we can (economically) store signatures for all the threats we encounter (regardless of whether we think that threat will become popular).   While traditional AntiVirus vendors could theoretically ship signatures for every threat they know about, doing so with the traditional push model is too cost prohibitive.  For example, if one were to try shipping signatures for all known threats, it would require (at least) on the order of a half a gigabyte of hard-drive space (something very few users are comfortable with).    In this regard, a cloud model is not only more economical and more efficient, but it leads to improved protection and overall product quality.

For all their benefits, though, it is clear that we shouldn’t put all of our eggs into the traditional signature basket.  Far too many threats have short lifetimes, and using traditional 1-1 signatures to catch each one not only fails to scale well, it also fails to adequately protect our users.  If a single threat can infect a system, it can wreak havoc and cause that system to be permanently vulnerable.  To address that concern, we’ve built technologies like Ethos and Spero, among others.  I’ll be describing those in my next blog post, so stay tuned!

Immunet Planned Upgrade/Outage

Dear Immunet User,

On Saturday October 9 (MST) we will performing upgrades to our Cloud Infrastructure. During this time Immunet Protect Free users will, from time to time, see that their status is 'Disconnected'. This outage is expected to continue through to the early morning hours of October 10 (MST). Not all regions will experience the outage at the same time.

Immunet Protect Plus users are still able to update their protections and will not be affected by this outage although their 'Connected' Orb will be in a red state until this upgrade is complete.

Best Regards,

Alfred Huger

VP, Development

Immunet Corp

Immunet's Core Design Principles and Protection Stack

Over my next few blog posts, I’d like to describe (at a reasonably high level) Immunet’s approach to dealing with today’s threats as well as our overall protection stack.   Even for a high-level description, there is a decent amount of material here, so I thought it would make sense to break it apart into several blog posts.  

In this post, I’d like to go into our core design principles.  Our goal in creating Immunet Protect was to build an AntiVirus solution with a clean slate leveraging what we understand about today’s threat landscape.   We found that many existing AntiVirus solutions were encumbered by legacy technologies and legacy approaches, which greatly inhibited their ability to meet the unique demands of current threats.  With that in mind, the following were the guiding principles used in our approach:

• Adapts rapidly:  Threats today are quickly changing.  We literally see many tens of thousands of unique malware variants each day.  These variants have a very short lifetime.  In fact, about 75%+ of threat instances that we have traditional signatures for only appear once in the field. Moreover, these variants are becoming increasingly complex -- with the current generation of viruses building upon their predecessors.  To make matters worse, in the fight against malware, attackers also tend to have an inherent advantage since they can download and install all the well-known antivirus solutions, and can keep morphing the threats they write until they evade detection.  On the other hand, AntiVirus vendors have less access into the operations of malware authors.  Consequently, AntiVirus technologies need to have a simple and rapid mechanism by which they can be calibrated to protect users in the field from these changes.
• Lightweight and interoperable:  All the well-known AntiVirus solutions available on the market today tend to be very heavy weight.  This shortcoming has several repercussions.  First, they tend to slow machines down considerably – perhaps even more so than the very threats that they claim to protect against!  Second, users tend to disable these heavyweight AntiVirus solutions.   Third, heavyweight antivirus solutions tend to have stability issues and even worse can have adverse behavior if running on the same system as other existing security technologies.   Our sense is that performance has become the number one criterion by which customers decide which solution to go with (even outranking the protection these solutions provide)!
• Data Driven:  Because of their current architecture, many existing AntiVirus vendors have limited insight into how their products are really performing in the field.   For a given threat, they may not know how many times it has triggered in the field, and for a given technology in their protection stack, they may have a hard time understanding its true in-field efficacy (as well as the specific value it adds).  Oftentimes when a new antivirus technique is deployed within an existing antivirus product, vendors are essentially shooting in the dark.  This approach can be costly since one has to literally deploy a technology into production before being able to determine how it will actually perform. While this approach could suffice for yesterday’s threats (which were more simple in nature), it is not appropriate for today’s more complex (and more unpredictable) threats.   
• Community Aware:  Threats tend to propagate across “social” networks.  For example, if your friend’s computer is infected, chances are that it will be a launch pad for an attack on your system.  For example, an attacker will compromise your friend’s system, and then send an email from your friend’s account containing malicious content.   Along similar lines, if you share a USB stick with a friend or colleague, then it can be used to carry a threat from their machine to yours.  The adage that there is safety in numbers is especially true for online threats.

The architecture of Immunet Protect, which we will describe in the next few posts, was developed with the above principles in mind.  

That file you just restored, might be Conficker!

At Immunet we constantly try to find the balance between fast, community based detections and 'false positive management'. In simple terms this means we spend a lot of time trying to hunt down threats in our community and try diligently to avoid convicting innocent files as malicious. Each day, 7 days a week, 3 times a day, we manually review our potential False Positive detections. We do this by looking at what files our users roll out of the Quarantine section in Immunet Protect. We examine each and every file. This allows us to correct errors we make and ensure strong in-field quality. Mistakes can happen, certainly, but we try to limit them as much as possible.

During this daily exercise we've found something quite alarming, many times a day we are seeing users rolling things out of Quarantine (in the Immunet Protect product) that are actually threats. Nearly every time this is because the threat is masquerading as software they really want to run (and it's often pirated...). In the last 72 hours 523 different Community users rolled back this SHA256 (and related threat name):

AA0BBAECB678868E1E7F57C7CA9D61B608B3D788BE490790EB1D148BEADF4615
Trojan.Rootkit-1503

It's our biggest single item rolled out of Quarantine in the last 72 hours. The rub is, this threat is real. In fact it's Conficker.

See the Virus Total Results.

This goes to show how insidious some of the packaging and social engineering can be to get people to run threats. Please continue to be careful about what you download and run off the Internet. Avoid running pirated software at all costs and be very leery of ANY links offered to you on Social Networking sites.