I have a home server running a desktop ubuntu distribution. I found this in my crontab

* * * * * /home/username/ /.access.log/y2kupdate >/dev/null 2>&1

and when looking in that directory (the space after username/ is a directory name) I found a lot of scripts which are obviously doing something they shouldn't.

Before I wipe that computer and reinstall stuff, I would like to find out what caused the security breach and when it was done. So I don't open the same hole again.

What log files should I look in? The only servers that I am aware of that are running on the computer is sshd and lighttpd.

What should I do to detect if things like this happens again?

  • @Person who voted to move to SU: How do you think virus/hacking forensics relates to Super User more than Server Fault?? – Chris S Dec 3 '10 at 14:45
  • 3
    I'm glad to hear that the plan is to wipe the computer and reinstall stuff. That is absolutely the right thing to do no matter what you find, because there's always a good chance there's something worse which you didn't find. – mattdm Dec 3 '10 at 14:51
  • Yep really should vote to move to IT Security (security.stackexchange.com), not SU or Serverfault! – Rory Alsop Dec 4 '10 at 14:11

First, make sure the computer is disconnected from any networks.
Second, make sure you get any important data off the drives before booting the hacked OS again.

Start with checking out the time stamps on the files in question. Often they are accurate.
Cross reference those with the httpd log and the auth log if they weren't wiped. If one other the other was wiped, you can bet that was the means of entry. If they're still in tact, you might be able to glean more information on how they got in from the log.

If they're all wiped, you're pretty screwed. It would likely take more time to figure out what happened than it's worth.

You mentioned those two services were running, was there a good firewall in place to prevent everything else from being accessed? Did you allow SSH on port 22; is your login reasonably easy to guess; did you allow password logins; did you have any sort of real rate limiting for password logins? Did you have any additional software installed with lighttpd; perl; php; cgi; a CMS or similar? Were you running updated version of all the software; do you subscribe to security notifications for all the software you run and carefully evaluate all notifications to see if they apply to software you run/expose to the public?

| improve this answer | |
  • The installation was a standard Ubuntu Desktop Edition 10.x, nothing in particular was done for security. I assumed there was a good enough firewall by default. Is that not correct? I allowed password login on port 22, but the password was a random 8 character string. Should be good enough? – Jonatan Kallus Dec 3 '10 at 15:27
  • I believe the default firewall is "Wide Open - No Firewall". Unless you carefully configured it, it's not doing anything. – Chris S Dec 3 '10 at 15:32
  • The question is what services were you running more than what firewall. Unless the firewall is set up to configure particular IP's for access, the best firewall is to not run unnecessary services in the first place. And what was your network setup? Open to the Internet? Inside a private network? Did you not have a firewall at the point where your internal network went into the Internet, or was your server in a DMZ? – Bart Silverstrim Dec 3 '10 at 16:28
  • The server was connected directly to the internet. I used it as a router too. If the default firewall is wide open, then I think I have enough explaination for how this could happen.. – Jonatan Kallus Dec 4 '10 at 13:01

This is kind of a topic in itself; you can google for linux forensics for more information. Basically you'd have to make an image of your drives for offline analysis first off, then wipe the computer and install from clean slate.

And remember all the incidentals. Anyone using the computer could have had their passwords compromised. Change passwords, keep it offline, etc. until you get it in a "clean room" (isolated VM).

Otherwise it's a lot of checking logs (which can be faked) and checking your applications (php scripts? Databases? Updated for latest fixes? other users giving out passwords?)

There's literally no easy way to answer your question since you'd need to do forensic work on the server and check for holes. You could use some automated tools but keep in mind if the attacker had Root privs you can't trust system binaries anymore, and you can't trust the logs.

As for future attacks, depending on how secure you want to make it, you can start by redirecting your logs to a system that is just used for saving system logs. No other access, to reduce the attack footprint.

You'd also run checksum software on your system like Tripwire to check the integrity of your files.

And of course keep up to date with updates and run scanning software that checks for rootkits.

Again, security isn't a throw-the-switch thing. It can be a specialty in itself too. Layered security can get as tight as checking for hosts/IP's that don't belong on your network, encrypting all access to the system, having daily logs of changes found on your system sent to you, and setting up a honeypot on your network to look for strange activity (why is my server trying to connect to port 25 on the honeypot computer?)

First and foremost if you want to check for activity, get the disk image and reinstall the server software. From scratch. The server's binaries can't be trusted anymore.

EDIT - some other things that occur to me since you're running SSH - install denyhosts. It can be configured so that automated attacks against your system on SSHD will be locked out after X number of tries. It can also be configured to update from other denyhost servers in a "cloud" to share locked out IP's to help minimize automated attacks. You can also move the port it's listening on; many people point out that it's just security through obscurity, but given the number of bots scanning, this cuts down significantly on random attempts to break in.

| improve this answer | |
  • Thanks! You really answered half of the question each, I wish I could mark both as the accepted answer. – Jonatan Kallus Dec 4 '10 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.