1

A linux server I manage is sending out hundreds of spam messages from a specific user account every 5 minutes.

I found, within the user's account (which runs Wordpress), a couple of PHP exploit scripts. One of these was the Meyhem dropper.

However, I can't find any signs on the system that the droppers have compromised the system. The files they reference don't exist, there are no running "host" processes, nothing in crontab, no unusual listening or connections on port 80, etc. I went through all running processes and didn't see anything unusual (may have missed something).

However the only sign of a problem is that the system sends out a ton of messages every five minutes, using the same "from" username. (the username of the compromised user mentioned above).

I can monitor /var/log/maillog and every 5 minutes these messages drop into the queue.

I can't figure out how to tell which process is doing this. I have stopped httpd, crond, atd, and disabled the customer's website so nothing would be hitting the malicious PHP files. I also renamed the PHP files. But still, every 5 minutes the log file shows another boatload of messages from this user being sent out.

Can anyone please point me in the right direction to find this exploit/malware/etc?

If there is a way to watch all newly spawned processes, I could see which one spawns at the time the spam goes out. Any ideas?

Excerpt from maillog:

Jan 28 18:12:13 xyz postfix/qmgr[6829]: C4BF3206075: from=<username@xyz.[domain name here].net>, size=1199, nrcpt=1 (queue active)
Jan 28 18:12:13 xyz postfix/qmgr[6829]: C0421227061: from=<username@xyz.[domain name here].net>, size=1222, nrcpt=1 (queue active)
Jan 28 18:12:13 xyz postfix/qmgr[6829]: C05082060C7: from=<username@xyz.[domain name here].net>, size=1180, nrcpt=1 (queue active)
Jan 28 18:12:13 xyz postfix/qmgr[6829]: BA6D922629A: from=<username@xyz.[domain name here].net>, size=1232, nrcpt=1 (queue active)
Jan 28 18:12:13 xyz postfix/qmgr[6829]: BC58A226B0F: from=<username@xyz.[domain name here].net>, size=1224, nrcpt=1 (queue active)
Jan 28 18:12:13 xyz postfix/qmgr[6829]: BB1C6227574: from=<username@xyz.[domain name here].net>, size=1216, nrcpt=1 (queue active)
Jan 28 18:12:13 xyz postfix/qmgr[6829]: B896B226EB3: from=<username@xyz.[domain name here].net>, size=1194, nrcpt=1 (queue active)
Jan 28 18:12:13 xyz postfix/qmgr[6829]: BC7532266B8: from=<username@xyz.[domain name here].net>, size=1186, nrcpt=1 (queue active)
6
  • post the maillog here
    – masegaloeh
    Jan 28, 2015 at 22:37
  • First, @HopelessN00b, I need to confirm that it is compromised. Then I have justification for wiping it. But as I stated, I can't find any indicators that would point to an infection. The only thing is the repeated emails being sent every 5 min. Jan 28, 2015 at 23:22
  • I just looked in the postfix deferred spool directories and found hundreds of files sitting there. I deleted all these and now the server is not attempting to send mail every 5 minutes. I'm thinking that during the time the compromised scripts existed on that user's wordpress account, they were using it as a mail relay. Jan 28, 2015 at 23:23
  • Looks like those messages queued up in 'deferred' and were possibly being retried every 5 min by postfix itself. Does that sound logical? Jan 28, 2015 at 23:24
  • well, you can start by issuing command grep BC58A226B0F /var/log/maillog to know, where the email enter postfix...
    – masegaloeh
    Jan 28, 2015 at 23:28

3 Answers 3

3

In php.ini find mail.log. Activate logging PHP mails.

mail.log = /var/log/phpmaillog

Also activate x-mail-headers

mail.add_x_header = 1

Reboot httpd to activate the PHP changes.

Than start monitoring this log.

tail -f /var/log/phpmaillog

Delete those PHP files. Install iThemes Wordpress plugin.

4
  • I just enabled mail.log, no entries showing up yet... mail.add_x_header was already set to "On". Jan 28, 2015 at 23:11
  • The mails just went out again. No entries appeared in /var/log/phpmaillog Jan 28, 2015 at 23:12
  • Also, I completely disabled the compromised user's account (website, email, DNS, etc) Jan 28, 2015 at 23:15
  • php -r "mail('[email protected]','test','qwer');" Can you do this on terminal? Please check if it writes in /var/log/phpmaillog.
    – borayeris
    Jan 29, 2015 at 0:17
1

If you also have a rootkit exploit then you will not be able to see the processes, network connections, etc. from the compromised machine. This is because the executables you use have been compromised and adjusted not to show the other compromised tools and services.

In this kind of situation you must, at the very least, boot from a verifiable separate OS (something like SystemRescueCD would be my starting point). Then you can verify all the executables against installation checksums and reinstall anything dubious. Ideally you would tear the system down and reinstall.

I'm sure there are many helpful articles about recovering from rootkits on https://security.stackexchange.com/

5
  • 1
    Does Linode offer any kind of rescue mode for systems that are borked and won't boot? (My own hosting provder does, hence the question.) Jan 28, 2015 at 23:21
  • Two problems: 1) this is a virtual server hosted at linode, so I can't boot it from any other media, and 2) I checked for many indicators of infection/rootkit per documentation and found none. Jan 28, 2015 at 23:25
  • YES it does! Cool, I had not thought of that. linode.com/docs/troubleshooting/rescue-and-rebuild Jan 28, 2015 at 23:26
  • What would be the most effective way to verify file integrity? Can I run a DIFF against a repository, or checksum important files, or is there something else that's more automated? Jan 28, 2015 at 23:26
  • You should be able to verify against your installation repository. It's not something I'd want to have to do, though, as it will take a lot of (your) time. HopelessN00b's reference to the "probable duplicate" article contains more detail on this and is worth reviewing. Jan 28, 2015 at 23:29
0

Perhaps TCPdump can give you an insight. Watch the NIC card sending traffic to that target i.e

tcpdump -i eth0 -p smtp -Z root (or add the IP, instead of SMTP)

Not the answer you're looking for? Browse other questions tagged .