The problem

A client of mine asked me to take a look at his shared-hosting webserver for the following problem, but I'm stuck at finding out what's wrong. His server is being blacklisted by a lot of major blocking list such as CBL, Spamhaus and the blockling list from Outlook.com.

What I've tried already

I started by looking at the users in his DirectAdmin environment but I didn't find any users whom are sending more than couple of e-mails per day. I downloaded his exim log, took a look at the mail queue, but couldn't find anything out of the ordinary. Next thing I thought of was running findbot.pl from CBL, but it came up only with false-positives.

Another thing I tried was to change the sendmail_path in php.ini to log every e-mail that is being sent out via sendmail. However, everytime I changed the sendmail_path, all PHP processes started to hang. I tried different ways (MailCatcher, my own scripts), but every change made the processes hang. Really strange, but after I few tries, I moved on to the next step.

Next step: installing lsofand create an bash script that would print the output of lsof -i | grep smtp into a log file, every second, while printing the outpot of ps auxw to another log file every second. This gave me some valuable information, but I can't track the issue yet.

Where I'm stuck

So after letting it run for a couple of hours, I opened up both log files and saw a bulk of this rules:

lsof - logfile

exim      10921    mail    9u  IPv4 2260427      0t0  TCP hostname-from-server.com:smtp-> (ESTABLISHED)
exim      10921    mail   10u  IPv4 2260427      0t0  TCP hostname-from-server.com:smtp-> (ESTABLISHED)

When I look at the logfile and search for the PID that is mentioned in the lsof logfile, I see the following lines:

ps auxw - logfile

mail      1750  0.0  0.0  59032  1320 ?        Ss   Nov28   0:01 /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
root     10909  0.0  0.0 103388   896 pts/2    S+   17:44   0:00 grep mail

mail      1750  0.0  0.0  59032  1320 ?        Ss   Nov28   0:01 /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
root     10917  0.0  0.0 103388   896 pts/2    S+   17:44   0:00 grep mail

mail      1750  0.0  0.0  59032  1320 ?        Ss   Nov28   0:01 /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
mail     10921  0.0  0.0  61112  1792 ?        S    17:44   0:00 /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
root     10923  0.0  0.0 103388   896 pts/2    S+   17:44   0:00 grep mail

mail      1750  0.0  0.0  59032  1320 ?        Ss   Nov28   0:01 /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
root     10931  0.0  0.0 103388   896 pts/2    S+   17:44   0:00 grep mail

mail      1750  0.0  0.0  59032  1320 ?        Ss   Nov28   0:01 /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
root     10939  0.0  0.0 103388   896 pts/2    S+   17:44   0:00 grep mail

The problem: there is nothing out of the ordinary with this line and I can't see which script, program of user called exim. When I take a look at the exim mainlog and rejectlog, I can't find the ip nor can't I find any line at all around 17:44 (time according to the ps auxw log).

When I follow lines from the logfiles from e-mails that I send myself, I can find them in the mainlog from exim at exactly the time that is mentioned in the ps auxw log. It appears that, somehow, the spammails aren't logged in exim or are removed immediately after sending.

My questions

  • I think I can solve my problem if I knew which script, program or user called the PID and invoked exim/mail. Does anyone have an idea?
  • Is it possible that some other server, not ours, is sending out spam and is, for example, spoofing our IP-address? Maybe this is a very dumb question, but I'm curious, since it so easy to spoof headers.

Additional information

Via the provider-portal of Outlook.com, we managed to get one of the e-mail headers:

X-HmXmrOriginalRecipient: someone-who-received-our-spam@hotmail.com
X-Reporter-IP: [IP-from-some-who-flagged-as-spam]
X-Message-Guid: a2236172-9474-11e5-9c3a-00215ad6eec8
Authentication-Results: hotmail.com; spf=none (sender IP is [OUR-IP-ADDRESS]) smtp.mailfrom=minvituccia@blackberrysa.com; dkim=none header.d=blackberrysa.com; x-hmca=none header.id=minvituccia@blackberrysa.com
X-SID-PRA: minvituccia@blackberrysa.com
X-SID-Result: NONE
X-Message-Status: n:n
X-Message-Delivery: Vj0xLjE7dXM9MDtsPTA7YT0wO0Q9MjtHRD0yO1NDTD02
X-Message-Info: 11chDOWqoTmjqhOzvWWho/vK8oL2x1FIoEm0Tn+r3D4Vy8IHo2wUnqS07yp2Fxclyw07ONZgeH1xFUrogbJOZz8Pfl5FrUXTGgolDal8+UhiPOrwCAKsLtRr0R42oH/Du2inmiSwuWc/pY9oiWRqLA5If7jw818pUulf3QP7m+wKn2HEVHAg2VBr+OqDk1w/hWWO68tIy1BSoE8QFSPMNXh31MYdKh4mif3jAqDU+0qWqWSAxPdE/A==
Received: from [our-hostname] ([our-ip-address) by COL004-MC2F4.hotmail.com with Microsoft SMTPSVC(7.5.7601.23143);
 Thu, 26 Nov 2015 11:34:05 -0800
Return-path: <minvituccia@blackberrysa.com>
Received: (qmail 18660 invoked by uid 61081); 26 Nov 2015 20:52:03 -0000
Date: 26 Nov 2015 20:52:03 -0000
Message-ID: <20151126205203.18660.qmail@our-hostname.com>
From: "Meghann Gasparo" <minvituccia@blackberrysa.com>
To: "someone-who-received-spam-from-our-server" <someone-who-received-spam-from-our-server@hotmail.com>
Subject: You could strike all your limpid seed right into my love tunnel text me 1.970.572.00.14
Mime-Version: 1.0
Content-Type: text/html
Content-Transfer-Encoding: 8bit
Mime-Version: 1.0
Content-Type: text/html
Content-Transfer-Encoding: 8bit
X-OriginalArrivalTime: 26 Nov 2015 19:34:06.0061 (UTC) FILETIME=[69C119D0:01D12881]

<html><body>Throw some of your hot cum on my face, deep into my door<br>
or <a href="http://holidayextravaganza.org/wp-content/themes/">run my humps rubbed</a> once again.<br>
<a href="http://holidayextravaganza.org/wp-content/themes/">Watch my profile</a> to receive much more spicy fun or just sms right now 1-970-572-00-73</body></html>


We don't use qmail as MTA. Needless to say, but the domain blackberrysa.com is not one of ours.

  • It doesn't have to be using exim. it's fairly easy to send mail directly. – user9517 Nov 29 '15 at 18:01
  • @Iain how can I check if that is the case? The PID (10921) indicates that it is using exim. – BlueCola Nov 29 '15 at 18:07
  • @Iain I just received one of the e-mail headers that is received by someone, from our server. Strange thing is, the headers says that the e-mail is being sent with qmail, but we haven't installed qmail on our servers. Any possibility that the spam isn't coming from our server at all, but someone is spoofing our IP-address? – BlueCola Nov 29 '15 at 18:23
  • 1
    Run maldet on all the users' files. You're bound to turn up lots of interesting stuff. – Michael Hampton Nov 29 '15 at 19:31

You could try looking at netstat to get outbound connections and filter for port 25

netstat -nptw | grep :25

You could use iptables to log outbound connections to port 25

iptables -I OUTPUT -p tcp -m tcp --dport 25 -j LOG --log-prefix 'Outbound SMTP connection'

This produces a log record like

Nov 29 18:15:56 hostname kernel: Outbound connectionIN= OUT=eth0 SRC= DST= LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=36829 DF PROTO=TCP SPT=39355 DPT=25 WINDOW=14600 RES=0x00 SYN URGP=0

Note the SPT=39355 you can use that to search your netstat output

netstat -anp  | grep 39355
tcp      0    1        SYN_SENT    13992/someprogram
| improve this answer | |
  • The rule is up and running! When I have more information, I'll update the post and place a comment. I also used the --log-uid option, as @Law29 suggested. – BlueCola Nov 29 '15 at 23:48
  • Another question, netstat is coming up with a lot of rules like this: tcp 0 0 [host-ip-address]:25 [destination-ip-address]:35597 ESTABLISHED 11932/exim. Exim is connecting to another port than 25, which is not being monitored by the iptables rule. Why is exim connecting to another port than 25? Is this (for example) for communicating with another server that just delivered an e-mail to our server? – BlueCola Nov 30 '15 at 1:13

You header indicates that the mail did not go through exim. The qmail "Received" header is a spammer fabrication. If you have a webserver on this server, the chances are that there is some compromised script. If you are doing NAT for other hosts, then the problem could be on the other hosts. Otherwise, you probably have a compromised server.

If you're hesitant, try

iptables -I OUTPUT -m state --state NEW -p tcp --dport 25 -j LOG --log-uid --log-prefix 'New SMTP Outbound'

This will log the user id of the process sending the mails.

| improve this answer | |
  • The rule is up and running! When I have more information, I'll update the post and place a comment. Thanks for your suggestion about the --log-uid, that's a smart one. – BlueCola Nov 29 '15 at 23:48

I recommend an inspection on the webserver(s) installation, where you'll probably find out lots of malicious PHP. Those PHP codes probably arrived to your webserver by someone/some-bot abusing one or more vulnerabilities of your server or its PHP files. Beware that those PHP codes may be found in many places and formats: inside a comment of a GIF or JPEG image, md5-summed, inverted, etc. Find those PHP functions and the evil eval and grep for them in places where you know they might not be there.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.