Has my Linux server been compromised? How do I tell? [duplicate]

Question

Running (X)Ubuntu 10.04.2 LTS behind a router.

I just received an email from my root account on that machine, with the following subject:

*** SECURITY information for <hostname>:

The message body contained this warning:

<hostname> : jun 1 22:15:17 : <username> : 3 incorrect password attempts ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/sh /tmp/tmpPHBmTO

I can see no /tmp/tmpPHBmTO file, though there is a file named /tmp/tmpwoSrWW with a timestamp dating from 2011-06-01 22:14, so just before the mentioned date/time. It's a binary file, and the content doesn't look familiar to me. Also, that file only has -rw------- permissions.

As I read it, this means that someone (or something) has (had) access to my machine. Apparently not root access (yet), but still, enough to write files to my /tmp directory at the very least.

Does someone have any pointers as to where I could look for more information: who could have done this, and how they could have done this?

My router is configured to allow access to forward traffic for SSH, HTTP (nginx acting as reverse proxy for one of several other services), SMTP, POP (postfix) and IMAP (dovecot), and also port 51413 (Transmission).

Answer 1

I realize it's been two years since the original question was asked, but in case somebody else comes here through Google like me: I saw that behavior caused by a Dropbox daemon running with a non-root account on a server which didn't run any X server. I managed to copy the files before they were deleted. For some reason the daemon wants to reset permissions on its data directory (never mind that it wouldn't even have needed root to do that) and kill some process. I can only speculate as to why, maybe it did some kind of auto update and tried to reload itself or something like that.

The file /tmp/tmpe1AGcd contains:

#!/bin/bash
sudo -K
zenity --entry --title="Dropbox" --text="Dropbox needs your permission to save settings to your computer.

Type your Linux password to allow Dropbox to make changes." --entry-text "" --hide-text | sudo -S /bin/sh /tmp/tmpAH5mxL
if [ "$?" != 0 ]; then
zenity --error --text="Sorry, wrong password"
exit 1
fi

Since the machine is running headless and the binary "zenity" isn't even installed, sudo receives empty password attempts and fails when trying to execute /tmp/tmpAH5mxL, containing:

#!/bin/bash
chown -R 1000 "/home/<username>/.dropbox"
chmod -R u+rwX "/home/<username>/.dropbox"
kill -s USR2 5364

The resulting message I got was the same you did:

<hostname> : Jul  4 16:32:24 : <username>: 3 incorrect password attempts ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/sh /tmp/tmpAH5mxL

Answer 2

If you have SSH opened to the Internet you WILL see hack attempts where scripties will try to crack that password.

Possible mitigation steps:

• Do not allow root login via SSH (su after login if needed)
• Have a VERY strong password (think passphrase - 10 or more characters)
• Use key authentication for SSH and turn off password auth
• Install fail2ban to block login attempts after n login attempts
• Move ssh off of the default port (as recommended by @voithos)

Note: if your machine is already compromised, none of the above will help.

Answer 3

This answer refers also to comments made on @uSlakr's post.

Your computer is at least partly compromised. Being able to write to your /tmp with your username indicates the ability to work under your user name. Being able to write under your username = able to do what you are currently doing.

If you have services that run under your username, mostly likely that particular service has been compromised. (1) If you do not have services that run under your username, then your account has been compromised totally (i.e. with interactive shell access). But this is less likely, see (1).

Your best bet is to (1) backup the current system to a virtual machine for inspection + restore from old backup (do inspect for files like that) + stop network access (2) change password of all account (3) disable all services (4) update all services to newest version (5) install intrusion detection program.

(1): As you can see the intruder is trying to run some program in order to obtain root access. It does not sound very much like they have interactive shell access -- if they had interactive shell they should be grabbing the hash of your password and crack it with some manner // installing some sort of program to your interactive shell to capture your root password // use sudo (these are generally better idea than having wrong password 3 times, well, if they are THAT stupid... nevermind)