Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Some days ago, the server of a friend had an intrusion. The attack installed a new SSH daemon that let any valid account in, without providing a valid password. After login, each account automatically got root permissions and the server greeted as follows:


The attack also removed the syslog entries that showed the intrusion (we figured that out with the log of the syslog) and changed the Debian package location path in /etc/apt/sources.list.

The server runs under Debian Etch and wasn't up-to-date during the attack: Apache/PHP didn't have all security updates installed. We think that the intrusion could have happen because of these missing updates, but we actually aren't sure. During the day before the attack, we installed Wordpress 3.0.1; but we don't know whether or not the installation of Wordpress was a door opener.

Is there any way to find out which security-leak on the server allowed the intrusion?

share|improve this question
The bad idea: that 'Skynet' is possible a web crawler + a pack of exploits (and if you get a list of exploits in public 'skynet' build, you have chance that a hacker behind the attack had used additional exploits to modify the pack). – kagali-san Nov 20 '10 at 22:10
up vote 2 down vote accepted

Once an attacker has gained root, it's often hard to tell how they got in because they can modify the system including logs. Unless you have logs going off to another system over the network, you may be out of luck -- particularly since you mention the logs being modified.

Obviously, the lesson learned is that you need to stay up to date on patches. We apply patches usually within a day of them being released via yum/apt-get.

Note that an attack against PHP, Apache, or Wordpress probably would have only given the attacker web-server privileges. But it seems from your description that they definitely got root. So if they did get in via a web app, there was another compromise to allow them to get from Apache to root. But I'm also wondering if they got in via a weak root password on SSH, etc...

However, knowing the method to attack it probably won't allow you to clean up from it. The machine is compromised very seriously at this point, from your description, and your best bet is to install a new OS from scratch and set the system back up. Audit all data and software you bring over from the old box. It is VERY difficult to fix a compromised box.

In any case, to answer your question about if there's any way to find out, here are some common places to look for what happened:

  • /var/log/syslog or /var/log/messages -- look for when the attack may have happened, possibly by gaps in the log, or activity you can't explain.
  • Other logs maybe they didn't wipe out like /var/log/security
  • .bash_history of various accounts including root and the web server. This may provide the commands they ran or information on what was done.
  • Look for processes that are running that you don't recognize, or processes that you do recognize that may be running from unusual directories or names. Look at /proc/$PID and /proc/$PID/fd of the various process IDs ($PID) of processes you see running.
  • Perhaps "last" will tell you something, but they probably wiped it out.
  • rkhunter can help tell you what is compromised on the system and help to find where you can look for more information.
  • Apache logs may show some indication of unusual activity, but it's often hard to filter out the attacks you get all the time from the one that actually broke the system open.

Much of this work is serious cross-correlation between different log files to find out what is interesting and what is just noise.

Doing this sort of investigation is not quick work. It's probably going to take 4 or more hours, especially if it's your first time doing one.

share|improve this answer
thanks for your answer – kraftan Nov 25 '10 at 8:34

You're probably out of luck unless you have some sort of remote logging.

I think it's best to chalk this up to "lesson learned", reinstall the system from scratch (seriously, do not skip that step!), and then keep the new system patched.

In my experience, it's far better to have automated updates and risk having to clean up after a bad automated patch than it is to do updates manually and risk getting behind and have this happening. In the first case, in the very rare event of a bad vendor/distro patch, you can usually fix things with a few minutes work at worst. With a system compromise, it's a whole rebuild and a lot of time lost.

share|improve this answer
thank you for your answer. – kraftan Nov 25 '10 at 8:34

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.