Ive always wondered, what can we do if a virus or worm happens to affect the main file server?

What if from the file server the clients get infected by said worm/virus?

How would you clean it up? Where would you even start in a compromised environment that has anti-virus in place?


"infects everything" as in files on a server, or everything as in 800 workstations are slamming the network with attempts to infect each other?

The "right" answer would be wipe and reinstall from backups. The practical answer isn't always so straightforward.

Most viruses now tend to be straightforward in that they're not infecting files, but infecting a few key files or acting as droppers, so your files aren't usually going to be spreading the malware. If you're hit by most popular self-propagating malware today there's usually a targeted disinfector available from popular AV sites. The hard part (usually) is getting to the site, as many of these malware programs will attempt to mask themselves and try disabling AV programs, DNS requests to AV sites, etc. so you'll end up looking for a way to get into the site to get the tool in the first place.

We've had large scale infections of our systems with a worm. The key for us was mitigation of risk. Out of 800+ systems, only a small fraction of them aren't running Deep Freeze, a program that restores computers to their original state when restarted. So for those systems we can use the "star trek" method of fixing the computers in the network. Shut down everything. All at once.

That left us with administrative systems, certain staff, and servers to repair. Many of them were already immune due to keeping up with patches. The others had a targeted disinfector run, then re-checked with a couple AV programs to verify they weren't showing signs of infection.

We also used tools to scan the network for systems that weren't patched or had remote signs of the infection (it was a worm that had a network signature with the right scan method) so we could target our efforts on what to triage for repairs. After all the signs of infection were off the network, we restarted all the Deep Freeze systems.

(secondary note - we also have outgoing port 25 blocked for all but our mail server to prevent getting our domain blacklisted)

So the best way to prevent that issue is to work on mitigation of risk, in our view. Students don't have profiles; makes it harder to spread downloaded (or drive-by) malware. Permissions segregate data in the home directories of servers. Deep Freeze prevents permanent infection on systems. AV helps mitigate risk, but we also have had (and still do) AV signatures that will kill legitimate executables due to a bad signature in the database somewhere, so the AV can be as big a pain in the arse as the malware itself. Firewalls are blocking access outside our network. Backups are in place to restore from bare metal if need be. Honeypots in the network can help detect oddball activity. Monitoring your switches and gateways for unusual activity can help. Updates on a regular schedule helps close vulnerable pathways of infection. And diversity is your friend...sometimes a Linux system or Mac can reach an AV site for grabbing tools when all the Windows systems are crippled. Linux systems are also fantastic for pulling special tools and scanners when searching for solutions on the network. It's saved my rump a couple times when troubleshooting things.

Our particular situation isn't necessarily typical, so mitigating risk is a plan you need to create specific to your environment. But that can be said about just about any risk mitigation system.

  • I would vote up, but dont have the points yet. Great answer! Thanks! – Jared Jan 24 '11 at 13:27
  • it is impractical to "rebuild" in this day and age even for small companies. It suggests taking the whole network down for days. And doing that would make most companies go belly up. Practical ideas like this one are much better. I was hired on contract to help clean up such an evironemnt - required a boot disk from the AV company to clean each PC, and some firewall rules to make sure certain traffic didn't leave. The environment had several hundred servers and 9000+ PC's. Even with 100 people and those boot disks, it took a couple of months to cleanup – cwheeler33 Jan 24 '11 at 14:06
  • Well I was assuming that you shouldn't get into this sort of complete infection situation in the first place. Antivirus and automated patching should help but also designing your network so that systems (and users) are naturally segregated. If you have imaging or system restore mechanisms in place already then a complete restore can be much quicker than sending people round with antivirus disks. But again you need to plan ahead for that rather than wait for it to happen and try to deal with the consequences. – JamesRyan Jan 24 '11 at 14:47
  • Rebuilding now is often impractical, but it's the answer for "making sure" since a lot of malware you won't know if it installed exploits or other code as a dropper. That's why layered mitigation is needed now. I wouldn't trust any AV disc/disk as a reliable total solution. There's too much change it can overlook something. Sometimes you have to figure out how much risk you're willing to take and go with the most practical balance between business needs and risk of infiltration. – Bart Silverstrim Jan 24 '11 at 14:53
  • +1 for blocking outgoing port 25. I learned that the hard way. – Nic Jan 26 '11 at 6:12

In most cases you would just wipe and restore from good backups.

  • For servers, yes. But what about for the rest of the network? How would you even turn off the "source"? Am i even making sense? Haha. – Jared Jan 24 '11 at 13:18
  • Most malware that's self-propagating turns your clients into infected sources...so there is no faucet to turn off, other than closing the infection vector. Get all uninfected systems turned off and disinfect then update and patch. – Bart Silverstrim Jan 24 '11 at 13:26
  • 1
    @Jared - "wipe and restore from good backups" applies to both workstations and servers. It's the only way to be sure. As Bart says, this maybe isn't always so practical (though it can be depending on how your systems are set up) but it still remains the gold standard. – Rob Moir Jan 24 '11 at 13:33
  • In most large networks where manual wiping would be unmanageable you would expect automated ghosting or restore checkpoint mechanisms to be in place. The really dangerous thing about manual cleaning is that you can't be sure if you have got rid of it completely or whether the obvious parts are just a smokescreen for something more underhand. – JamesRyan Jan 24 '11 at 14:51

In a situation where everything is infected, the first thing to do is disconnect everything from the internet. In a typical enterprise, workstations shouldn't contain anything but profile data, so restoring those with images should be easy. If you don't have the option, and cleaning is the only option, you are in for a treat. You will want to figure out what the infection is, and the treatment options. Go slow, you will want to have everything back to normal as quick as possible, but that's where mistakes happen. There isn't anything worse than thinking you have a set of machines cleaned, only to revisit them later to find out they are re-infected.

In the case of a main file server, the best course of action is to image the server to separate disks and set aside (if you have the resources) and wipe it, install fresh and restore from your backups. Without doing that, there really is no way to know if it was rooted and won't come back to bite you down the road.

  • Thanks for the great answer bud! Got what I was asking for. A shut down and one by one inspection seems to be the way to go on a non-imaged non redirected folders network. – Jared Jan 24 '11 at 13:28

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.