3

I've inherited a web server that's been compromised. Trying to figure out why apache was hanging and the causes of high server load, I found several copies of perlbot 4.5 in /tmp. I'm now trying to figure out how they got in the machine so I can close the hole(s). Ive been looking at various scanners, nessus seems nice, and I ran a scan on the machine and one of the websites hosted. But there are a couple hundred sites, too many for anyone to know the ins & outs of all of them, and I'm new here so I really have no idea what they might be doing. Is scanning each site the best option?

How would you check so many sites on the same machine for issues?

EDITED TO ADD: we are wiping everything and restoring from back ups. Which is good but still leaves us open to the original vulnerability. Scan each site one at a time with Nessus or Metasploit to try to figure out what that vuln is?

EDIT 2: It was phpmyadmin. Even though that would have been something I would have upgraded as soon as I noticed we were running it, I found out the problem specifically by pouring through apache logs. nessus and metaspolit were neat but not helpful. (I may not understand how to fully utilize them though, I just ran basic automated scans).

Improve this question
6
  • serverfault.com/questions/218005/… serverfault.com/questions/6190/…
    – Andrew
    Jan 24, 2012 at 2:47
  • Thanks, thats post is where I got the idea to try nessus and acunetix from.
    – karmet
    Jan 24, 2012 at 4:01
  • 1
    it sounds like one of the user accounts has been compromised, but which user owns the perlbot files in /tmp? if they are apache:apache, or ftp, or some user account then that is a clue
    – Tom
    Jan 24, 2012 at 17:58
  • In addition, if someone has guessed a password for a ssh/ftp user then nessus and metasploit won't help, because the things you are seeing are attempts at escalation attacks from a local login. Look at the lastlog file, if that has not been blatted, or other root controlled files are intact then typically the problem was limited to non-root. (you still need to rebuild)
    – Tom
    Jan 24, 2012 at 18:03
  • But the hacker is writing stuff to /tmp because it is world-writable, if they were root then expect much worse like kernel and lib changes, which you wont be able to detect normally.
    – Tom
    Jan 24, 2012 at 18:10

4 Answers 4

Reset to default
0 
find / -mtime -1

Replace -1 with the number of days since when you think the intrusion happened. By finding the files that were modified, you have a fairly good chance of figuring out which site was used to get in. For instance, look for uploaded backdoor scripts in image folders and things of that nature.

Improve this answer
3
  • This may be the accepted answer, but it's a bad solution in this case. Who's to say that for example the find binary was not modified? You think that doesn't happen? The only fix is backup, wipe the server clean and rebuild. As suggested below. Anything less drastic is asking for trouble.
    – aseq
    Jan 26, 2012 at 21:51
  • This was the technique that I used to solve the problem.
    – karmet
    Jan 27, 2012 at 4:09
  • @kswift, so just for completeness, can you update with the cause and extent of the intrusion...? ie what what was the vulnerability, what was the intrusion method, and how did you remove the payload?
    – Tom
    Jan 27, 2012 at 16:37
7

Backup the sites, rebuild the server, then add back only the files that you know are needed and That do exactly what you expect.

It is actually less work than attempting to clean up a compromised system.

If you are dead-set on finding vulns, patch the server and then run metasploit against it. I'd also watch netstat to see what connections are being made. I'd also disable any accounts that don't belong there. Check what is started at boot for oddities. The list goes on...

http://sectools.org/ has a list of their top 125 security tools, which you can filter somewhat. This may give you some assistance choosing a tool, but for this you are going to have to do your own research.

The last time I needed to use metasploit or nessus was years ago, so I don't remember the specifics of using them anymore.

Improve this answer
4
  • Thanks for the suggestions. I'll look into metasploit. It is something that I will have to run against each site individually?
    – karmet
    Jan 24, 2012 at 4:03
  • (also I would prefer to rebuild vs clean up, and rebuilding might still be an option but the decision isnt mine)
    – karmet
    Jan 24, 2012 at 4:20
  • Tell the powers that be that nuking from orbit is the only way to be sure. (Bonus points if you directly quote Ellen Ripley.)
    – gWaldo
    Jan 24, 2012 at 5:08
  • After the rebuild, run the distribution package update yum -y update or apt-get update etc, and then configure tripwire package to monitor your binaries "tripwire --init"
    – Tom
    Jan 24, 2012 at 17:42
1

Given that you have decided to rebuild the server, as a post-mortem you can do the following;

Make a READ-ONLY Image of the bad machine and store securely

Make a copy of the filesystem on the offending server. To do this you will need another drive or mounted volume with sufficient space; Look into the dd tool. How you use it will depend on whether you have a single big root "/" partition, or lots of /var /usr and /lib filesystems etc.

Conduct forensics in a secure sandbox environment

Move the copy of the offending image of the file-system to a secure location, such as a guest ubuntu in a VirtualBox. Specially prepare the sand-box virtualhost. Install tools such as sleuthkit, foremost, clamav, autopsy.

Conduct the post-mortem in a safe and sensible manner

Don't give the sandbox access to the internet or local drives. Delete the sandbox guest after the process is over. Make notes, document changes.

Improve this answer
2
  • I agree with your methods (hence a +1) if you're interested in forensics (the how), but it sounds to me that he's more interested in just getting clean and back in business.
    – gWaldo
    Jan 27, 2012 at 15:46
  • @gWaldo Point taken, I also added some pointers to the question on determining the intrusion point. I guess the question has slightly evolved with the situation.
    – Tom
    Jan 27, 2012 at 16:36
0

The first thing to do is take all the sites offline. If your sites are compromised, they are probably being used to host nefarious files and/or to send spam.

The second thing is to alert any transactional banking services that any of the sites use.

The third thing is to alert your user base. These three steps can be done under an hour and there is no excuse if you haven't done so already.

Improve this answer
1
  • While all that should be part of the process you have completely failed to address the actual question.
    – John Gardeniers
    Jan 24, 2012 at 9:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .