I'm trying to compile a procedure for when ever there is a suspicion, perhaps from netflow data or something like that, that a server could be compromised. The logic being, that you should think about these things coolly and calmly before they happen for real and the pressure is on and the adrenaline flowing.

Can anyone recommend some good resources I should familiarise myself with to help me draft a sensible procedure?

I'm looking for good information on what to do to confirm suspicious, as well as what to do when a hack is discovered.


You can run something like chkrootkit on a schedule. http://www.chkrootkit.org/ Same for rkhunter.

Look for suspicious processes and shutdown unnecessary daemons.

If this is an rpm-based system, you can look at the output of rpm verify (rpm -vVa) to look for changes in installed packages.

There's always Tripwire... http://tripwire.org/

  • Why would you trust the output of a command running on a possibly compromised system? At the very least you'd have to bring the system offline and boot from known good media. I also hate the very concept of rkhunter and others. It's much easier to verify known good than it is to look for signs of badness. – Alex Holst Oct 27 '11 at 22:51

Assuming that you're explicitly not addressing the issues of preventing a compromise, nor the issues of recovering from a compromise....

Before it happens

You need to have secure logging in place - i.e. log data should be immediately published to a separate box for recording.

You should have a host based IDS to detect unauthorized changes - again with data storage off the box - such as tripwire / LIDS

You also need to plan for what you're going to do immediately you suspect a compromise - have you got a seperate unit you can swap in? If its a straight copy, then it will have the same vulnerabilities as the box it is replacing. Can it be configured to provide a reduced service with better security (e.g. a webserver with a read-only filesystem and bare-bones content).

Decide criteria for involving law enforcement. If you may be involving them - and they are likely to be interested - go speak to them in advance and ask how you can make their life easier.

Get agreement from all the stakeholders to the planned response.


In addition to basic anomoly detection, you should be checking the output of the intrusion detection system, and running rootkit checks regularly, also running frequent port scans against the box. Your routine anomoly checking should include log analysis.

While the methods described above are of value where the system is modified by the attacker, they do not address the problem of information disclosure. AFAIK the only sensible way to address this is via honeypot data (e.g. email addresses, user accounts).

when it happens

Pull the plug out. Seriously. A system shutdown may make significant changes to the system. You want it disconnected from any other devices on the network as soon as possible - but you need to preserve as much as possible about the state of the system.

If you're going to involve law enforcement - let them know before you do anything else.

If you want to investigate yourself, boot up the system from a USB / CD - NOT from the installed OS.

  • +1, especially for centralized logging and pulling the plug out. If I would be an evil dude, I would replace reboot/shutdown commands with something that first would modify the system somehow (like erasing the logs and shell history) and only then perform the restart/shutdown. – Janne Pikkarainen Oct 27 '11 at 18:40

You won't determine this just by looking at netflow output, since that is not nearly detailed enough to tell.

Instead, I would suggest using netflow suspicions as a starting point: look at it for some time, try to see if it is trending up or down, and if you suspect there may be service outages caused by it, pull the plug on the affected machine(s).

You do not perform forensics on live boxes, ever; if the suspicion becomes acute enough, you pull the box off the network, and then examine the fallout at your leisure.

  • Not just the off network... pull the plug! books.google.com/… It's your best chance at getting an accurate follow up investigation. – Chris Nava Oct 27 '11 at 14:38
  • "if the suspicions become acute enough" - that's the bit I'm asking about! What do you do to get from 'huh that's strange' to 'kill the service sod the users, this is serious'? – Bart B Oct 28 '11 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.