Anti-detection and anti-analysis techniques of modern malware – are they stumping security researchers?

4 10 2012

Modern malware have been evolving to such an extent that it is difficult for security researchers to keep up with them. Most anti-virus solutions depend largely on signature-based scanning and some form of heuristic-based detection. However, there is no doubt that creating signatures for every single malware is similar to a catching game where the good guys are always behind. The total number of unique malware variants jumped from 286 million in 2010 to 403 million in 2011 [1], a staggering 40% increase. For every new virus or worm that is discovered, many machines would have already been infected while the anti-virus vendors are still frantically trying to analyze the malware and release new definitions to their customers.

Malware writers are aware of this fact and have been incorporating advanced techniques to further delay the detection and analysis of their malware. One of the most commonly used methods is polymorphism, whereby the malware constantly mutates its own code to bypass signature-based anti-virus scans, but still retaining the malicious payload. This is typically achieved through encryption with varying keys, compression and filename changes [2]. Although polymorphic malware have been around since the early 1990s, the good news is that most modern anti-virus software can detect them with a decent probability [3] – usually by looking at the portion of the malware that does not change. However, cybercriminals have already found a way to bypass this – through a variation called “server-side polymorphism”. In this case, the malware is hosted on the attackers’ server, allowing them to generate a unique version for every potential victim each time the malware is downloaded. [4] This essentially renders pattern-matching useless since no two version of the malware is the same.

More alarmingly, a new form of attack called Domain Generation Algorithms (DGA) has manifested itself in new-generation malware. Popularized by the Conficker worm in 2008, DGA malware contains code that allows it to receive commands from remote locations. [5] Each day, the malware will generate a new list of domain names and try to contact all these locations for an update. Since the malware author knows the algorithm, all he needs to do is register one of the domains and host the update on that site. This makes the job of cyber law-enforcement officers extremely difficult since they have to shut down all possible domains to prevent the update while the attacker only needs to use a single domain. [6] Furthermore, signature-based detection will be irrelevant since the update from the remote site will be able to modify the malware source-code and behavior. [7]

In response, malware researchers have turned to advanced techniques to analyze modern viruses. This includes sandbox testing, emulation and using virtualization technologies. They confine the malware to a restricted environment so as to limit its actions, allowing a more effective analysis and reverse-engineering of the malware.

Unfortunately, it seems that even so, cybercriminals have found ways to circumvent the analysis techniques of researchers. Recent years have seen a rise of anti-virtual machine malware (or VM-aware malware), which can distinguish whether it is present in a virtual machine or a real environment. If such malware recognizes that it is being run in a VM, the malware will feign benign behavior and not release its payload. [8]

Like a cat-and-mouse game, the white hat community has yet again formulated possible solutions against such malware. In the recent 2012 BlackHat Conference, three researchers presented their findings on anti-VM, anti-debugging and anti-disassembly techniques used by malware. [9] They analyzed more than four million malware samples and in doing so, created a malware sample database with an open architecture. This allows other researchers around the world to see the results of the analysis, as well as develop and plug-in new analysis capabilities.

Some experts have also claimed that the more evasive a malware tries to become, the greater chances that it will gain unnecessary attention due to its over-innovative methods. [10] This is because even with anti-VM and anti-debugger features, researchers thus far have still been able to bypass the evasion techniques and deconstruct such malware.

However, it is only a matter of time before malware authors invent new ways to work around the efforts of security researchers. It is true that the white hats might have the right solution for a while, but there can be no silver bullet for malware, or any issue pertaining to the IT security industry. The only solution is to be ever vigilant and be as agile and adaptive as the black hats are.


[1] Symantec Internet Security Threat Report 2011. Rep. no. 17. Symantec, Apr. 2012. Web.

[2] Rouse, Margaret. “Polymorphic Malware.”, Apr. 2007. Web. 04 Oct. 2012. <;.

[3] Cluley, Graham. “Server-side Polymorphism: How Mutating Web Malware Tries to Defeat Anti-virus Software.” Naked Security, 31 July 2012. Web. 04 Oct. 2012. <;.

[4] See [1].

[5] Markoff, John. “Worm Infects Millions Of Computers Worldwide.” The New York Times. The New York Times, 23 Jan. 2009. Web. 04 Oct. 2012. <;.

[6] Constantin, Ucian. “Malware Authors Expand Use of Domain Generation Algorithms to Evade Detection.” IDG News Service, 27 Feb. 2012. Web. 04 Oct. 2012. <;.

[7] Ollmann, Gunter. “Domain Generation Algorithms (DGA) in Stealthy Malware.” Domain Generation Algorithms (DGA) in Stealthy Malware «. Damballa, n.d. Web. 04 Oct. 2012. <;.

[8] Sun, Ming-Kung, Mao-Jie Lin, Michael Chang, Chi-Sung Laih, and Hui-Tang Lin. “Malware Virtualization-Resistant Behavior Detection.” 2011 IEEE 17th International Conference on Parallel and Distributed Systems (2011): 912-17. IEEE. Web. 4 Oct. 2012.

[9] Branco, Rodrigo Rubira, Gabriel Negreira Barbosa, and Pedro Drimel Neto. Scientific but Not Academical Overview of Malware Anti-Debugging, Anti-Disassembly and Anti-VM Technologies. Tech. Qualys – Vulnerability & Malware Research Labs, 2012. Web. 4 Oct. 2012. <;.

[10] Mushtaq, Atif. “The Dead Giveaways of Vm-Aware.” FireEye, 27 Jan. 2011. Web. 04 Oct. 2012. <;.




Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: