Possible Duplicate:
How to find out if my server is compromised?

Recently I received a "Undeliverable Mail Returned to Sender" message from a .ru domain referring to a message that I'm sure wasn't sent by me or any code I have written. I'm using Google Apps for email and the server in question is a vanilla Ubuntu 10.04.1 server on a cloud hosting provider. I'm looking for a checklist of things I can do to determine if my server has been compromised. So far I've run the domain through an Open Relay test (it's not an open relay), and run netstat -an and ps aux to determine that only the programs that I want listening to ports are running (SSH, Tomcat, MySQL, Postfix (I don't think I can disable Avahi because of how my cloud service provider works)). And I've run cat /etc/passwd to make sure there aren't any unknown users. Is there anything else I should do?

  • Thanks for the link. From this I've added grep nmap /home/*/.bash_history and tightening my iptables to my list of things to do on my new server. Mar 9 '11 at 18:46

The mail you received may very well be "backscatter", that is, the result of spam sent using your email address and the .ru SMTP server bouncing it. Not necessarily because your server was compromised and the email sent from it.

  • 1
    This is probably the first thing to check before you start panicking. If the bounce included the original email with headers, see if there was a Received: header with your IP. If not, then the spammer forged your email address. DKIM and SPF help with this kind of issue but relies on everyone else's email server checking to see if the sender is forging the address.
    – DerfK
    Mar 9 '11 at 18:37

Run a sniffer such as wireshark and look for suspicious connections, but I don't think you've been compromised.

It's likely that someone is simply spoofing your email address as a "sender". It's trivially easy to do this. As they send spam out any rejects will come back to you (aka backscatter). Many email servers will return the email with full headers in the NDR (bounceback) so you can determine if this is the case.

If you do find that the suspicious mail is originating from a server that's not yours then I'd recommend adding sender authentication to your email relay. In this case you'd want your record to refer to google's spf record.


  • Thank you for suggesting something to try. While @jliendo's answer was technically correct and first, this is the kind of stuff I was looking for. Good luck building your SF score! I really appreciate all of the help. Mar 9 '11 at 18:43
  • You're welcome Jason. Glad I could help, score isn't important.
    – Ryan
    Mar 10 '11 at 18:41

Step one is to see if someone's spamming from your server.

First go through your mail logs! I'm used to exim, but I'm sure the steps are the same. Look in your mail queue, and see if there are any messages queued. A large spam run will always leave you with messages that can't be delivered for days at a time, because spam mailing lists always have invalid entries.

Also parse the mail logs. For exim I use grep '<=' /var/log/exim_mainlog | awk '{print $5}' | grep \@ | sort | uniq -c | sort -nk1 to see how many messages various people have been sending.

Lastly, check the headers in the bounce messages to see if the mail is actually coming from one of your machines/instances. You could just need SPF records.

Chances are by now you'll have a good idea whether or not your server is hemorrhaging mail everywhere.

If you've found spam messages, try to see what mail user they're being sent as. Change that user's password. Usually random spam comes from php scripts, php usually includes an X-Script-Sentby: (or something) header in the messages full headers, that you can use to find the responsible script.

Most of the time you'll find some random php script that fires mail everywhere. Occasionally I see people spamming through horde (which is a little harder to track) or outlook or using other clients that just require a password.

If mail is coming from your server and you're not sure where, you can also put disable_function=mail in your php.ini

Being rooted is the least likely scenario, more likely is some random php application got hacked. If you did get rooted try rkhunter or something, also hackers tend to use bizarre names for script, and php scripts left by hackers tend to be eval(base64_decode(gunzip( compressed, so if you grep -r eval $docroot you can usually find malicious scripts. Lastly, clamav isn't too bad at picking up malicious php hacking scripts.


Without physical access to the system or the infrastructure surrounding it you don't have a lot of options. The first assumption is that any tool on a compromised system may also be compromised. These days attacked systems will normally have a root kit or remote control software installed (I've been out of the game for a while) that will make it much more difficult to identify and often times subvert common operating system utilities and commands.

1) Port scan the system using a external utility 2) You can try moving over some statically linked binaries (additional copies of ps, netstat, etc) and using those to verify ports, accounts, and processes (statically linked means it's not going to use anything external to the binary (like a shared library from your system).
3) Have you considered someone may have sent a message using a return address that points to you? (backscatter...there's a term for it!)

Not the answer you're looking for? Browse other questions tagged or ask your own question.