With the recent problems that McAfee customers have had over the last week there has been lots of opinion that not only should the AV vendors have better testing but customers should test AV signatures before deploying.

Is this feasible? If you are doing this already do you take other measures to minimise exposure to malware while you are testing?

Update: With respect to feasible I mean is it feasible to run through a whole gamut of tests for each AV signature/definition update or do you restrict to just critical apps or critical functionality tests? Do you have a different approach for servers compared to desktops?


It shouldn't be much trouble setting up a test environment that gets the definitions first (by an hour or whatever is possible with your AV vendor) and whatever AV alerts it generates halts company-wide deployment until someone confirms it.

Setting this up for manual testing is bound to fail as someone said, due to boredom, not because it's not possible.

But just to include some perspective in the "debate", when talking Windows, switch to something like Software Restriction Policies or in Windows 7 / 2008 R2, the much improved version called AppLocker - it takes the more mature approach of only allowing specified programs to run, not the other way around...

...you can do this with code certificates too so say all Microsoft applications are allowed, and internally developed applications are allowed - without having to specify individual executables.

When properly implemented, no AV software is needed.

I can think of no firewalls today that starts out with "allow everything except specified malicious traffic". Unless someone did a poor job, they're all "block everything except specifically allowed traffic".

  • 'when properly implemented' will never happen because you simply can't remove all the bugs and attack vectors, just look at how completely locked down systems like consoles are hacked to run user generated code. All this really does is hurt the legitimate user and increase support costs in responding to their changing requirements – JamesRyan Apr 26 '10 at 9:19
  • How does not allowing users to run arbitrary code "increase support costs"? Have you tried AppLocker in a well-defined environment? It practically configures itself. Sure, if you don't have each system put down with owner and administrator and documented change routines this will be hell, but if you have those other problems, then need to be dealt with first anyway. I just think this is an approach that many find uncomfortable because it's a different approach, not because it would be harder to implement per se. If users cannot install random software, why let them run random software? – Oskar Duveborn Apr 26 '10 at 9:48
  • It adds support effort for us because of the politics of users who have laptops and request arbitrary programs to be run that are inherently unsafe and in our environment, we can't reject the requests, and we have unhappy users if the laptops don't work the way they're expected to when the users are out and about, and we're already understaffed for the number of people we support now. Some environments probably wouldn't have this problem if they have the authority to blanket-lock everything to virtual kiosks. – Bart Silverstrim Apr 26 '10 at 10:01
  • Well obviously this only applies to locked-down environments where users aren't in possession of administrative access to their work tools and management already has approved sane IT policies. White-listing is just a tool that enforces a policy much better than AV black-listing, but first you have to have that policy and work-flow support. Maybe your environment doesn't even work that way, what do I know, but there sure are environments that do. – Oskar Duveborn Apr 26 '10 at 12:06
  • @Oskar: No doubt locked down and highly regulated environments are out there (I hope my bank is one of them...) but I really think they're outnumbered by environments that aren't ;-) – Bart Silverstrim Apr 27 '10 at 9:57

Using McAfee's own ePo (ePolicy Orchestrator) AV client management server you can do just this - by a combination of scheduling when you download the Dats from McAfee and setting your test machines to get the updates before the general population you can try out the dats for a day or 2 before deploying them. I imagine most "enterprise' AV solutions will have similar mechanisms.

I have 2 problems with this however:

1) You will get bored - 1 single dodgey dat out of several thousand in what 5-6 years?
It comes down to a simple cost benefit equation - if the several hundred "man hours" of testing every year cost less than the profit loss caused by a loss of service from a dodgy update (Coles Supermakets for example), then of course do it.

2) More importantly it is really really important to get new definitions out to your users as soon as possible. To illustrate; just last week I was handed a USB drive infected with an Autorun type malware - this wasn't in the current dat and was picked up with the next days set of definitions (I only detected it as I was using a Mac and saw the suspicious files).

Fundamentally McAfee should never have released a DAT that had a false positive of a Windows system file! We pay McAfee a significant sum of money every year and expect their quality control to be better than this!


As an aside, did anyone else think of this xkcd when they saw cash registers running AV software...


The thing to remember is that the longer you wait to deploy the definitions, the longer the window of opportunity for new infections to take root.

The AV vendor is supposed to be testing things and McAfee acknowledged that they screwed up their internal tests.

In order for you to test it, you'd have to have a simulated environment that is running a machine of each of your deployments with the exact dll installation, application combinations, update combinations, etc.

...so basically you'd probably not have a guarantee that you'll catch edge cases.

BUT you can control the effect by using backups. You may not be able to stop disasters from happening, but you can control the outcome; having backups available means you can get them back up and online, and you may be able to roll back changes (usually, the latest McAfee issue was a fluke that shot Windows square in the brain that time, but once people knew what caused it at least the file could be copied over with a Linux boot disc from the sound of it...)

So in the end you're asking to duplicate a lot of work with little payoff a more risk to your users. You'd be better off making sure you keep periodic backups so you can restore your user systems and protect their data.


If you rely primarily on heuristic detection rather than signatures then there are fewer update. There are therefore fewer updates to test. I'd like to think we're slowly moving in this direction.

With that basic heuristic detection in place, you have more time to test the updates. And you can quite easily run those updates on a Continuous Integration or build server like developers do. However that'd take quite a lot of infrastructure, especially if you're testing against every possible setup you have running in your enterprise.


You could stage to a subset say 10% on a much more frequent basis. If your phone starts ringing, well at least only 10% of your computers have a problem. We used this method for deploying windows updates, we would do 10% for 2 days and then the rest of the computers. You could also find an av vendor that has a better reputation in not hosing your systems.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.