Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

What are main steps doing forensic analysis of linux box after it was hacked?

Lets say it is a generic linux server mail/web/database/ftp/ssh/samba. And it started sending spam, scanning other systems.. How to start searching for ways hack was done and who is responsible?

share|improve this question
add comment

8 Answers 8

up vote 10 down vote accepted

Here are some things to try before rebooting:

First of all, if you think you might be compromised unplug your network cable so the machine can't do further damage.

Then, if possible refrain from rebooting, as many traces of an intruder can be removed by re-booting.

If you thought ahead, and had remote logging in place, use your remote logs, not the ones on the machine, as it's all too easy for someone to tamper with the logs on the machine. But if you don't have remote logs, examine the local ones thoroughly.

Check dmesg, as this will be replaced upon reboot as well.

In linux it is possible to have running programs - even after the running file has been deleted. Check for these with the command file /proc/[0-9]*/exe|grep "(deleted)". (these disappear on reboot, of course). If you want to save a copy of the running program to disk, use /bin/dd if=/proc/filename/exe of=filename

If you have known good copies of who/ps/ls/netstat, use these tools to examine what is going on on the box. Note that if a rootkit has been installed, these utilities are usually replaced with copies that won't give accurate information.

share|improve this answer
    
The problem is how to find out if your copies of ps/ls/... are good. You could check their md5sum, but then again, md5sum may have been replaced as well. –  amarillion May 13 '09 at 14:15
2  
I keep a second copy of those critical files (and md5sum), along with the md5sums of the originals on all our systems. Then I have nagios manually check their md5sums for a match every hour. –  Brent May 13 '09 at 15:52
add comment

That totally depends on what was hacked, but in general,

Check the timestamps of files that were modified inappropriately, and cross-reference those times with succesful ssh (in /var/log/auth*) and ftp (in /var/log/vsftp* if you're using vsftp as server) to find out which account was compromised and from which IP the attack came.

You can probably find out if the account was brute-forced if there were a lot of unsuccessful login attempts on the same account. If there were no or only a few failed login attempts for that account, then probably the password was discovered in some other ways and the owner of that account needs a lecture on password safety.

If the IP is from somewhere nearby it could be an "inside job"

If the root account was compromised, of course you're in big trouble, and I would, if possible, reformat and rebuild the box from the ground up. Of course you should change all passwords anyway.

share|improve this answer
add comment

You have to check all the logs of running applications. For example, Apache logs may tell you how a hacker could execute arbitrary commands on your system.

Also check if you have running processes that scan servers or send spam. If it is the case, the Unix user they are running from can tell you how your box was hacked. If it's www-data then you know it's Apache, etc.

Be aware that sometimes some programs like ps are replaced...

share|improve this answer
add comment

You should ask yourself first: "Why?"

Here's are some reasons that make sense to me:

  • Assess the damage
  • Figure how they got in
  • Determine whether it was an inside job

Going beyond that often doesn't make sense. The police often don't care, and if they did, they would impound your hardware and do their own forensic analysis.

Depending on what you find out, you may be able to make your life alot easier. If an SMTP relay gets compromised, and you determine that it was due to a missing patch exploited by an outside party, you're done. Reinstall the box, patch whatever needs patching and move on.

Often when the word "forensics" is brought up, people have visions of CSI and think about figuring out all sorts of excruciating details about what happened. It can be that, but don't make it a huge lift if you don't have to.

share|improve this answer
    
It is my employers policy to investigate. –  Kazimieras Aliulis May 14 '09 at 6:52
    
From a sysadmin's point of view, forensics is not about blaming or legal issues, but about patching and better security –  Brent May 23 '09 at 13:58
add comment

Naaah!

You should shut down, connect the hard disk to a read only interface (it's a special IDE or SATA, or USB, etc... interface that doesn't allow any writes, something like this: http://www.forensic-computers.com/handBridges.php ) and do an exact dupe with DD.

You may do it to another hard drive, or you may do it to a disk image.

Then, store in a profer and totally safe place that hard disk, is the original proof without any tampering!

Later, you can plug that cloned disk, or image in your forensic computer. If it's a disk, you should plug it through a read only interface, and if you are going to work with an image, mount it 'read only'.

Then you can work on it, again and again without changing any data...

FYI, there are "hacked" systems images on the internet for practice, so you can do forensics "at home"...

PS: What about the hacked system bringed down? if I think that system is compromised, I wouldnt leave it connected, I would put a new hard disk there, and restore a backup or put a new server in production until the forensics finish...

share|improve this answer
add comment

I havent read the other responses, but I would make a ghost image of it to preserve the evidence and only examine the image....maybe...

share|improve this answer
add comment

I highly recommend reading "Dead Linux Machines Do Tell Tales", an article by The SANS Institute. It's from 2003, but the information is still valuable today.

share|improve this answer
add comment

Take a memory dump and analyze it with a memory forensics tool, such as Second Look.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.