Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

My server sent 83,000 spam emails last night, I've been trying to track down the culprit, but I'm not sure how to find out exactly.

  • in the logs the "from" address is always something like @#!
  • the connections appear to be from the localhost

leading me to believe this is a script using the php mail(); function or a CGI.. so, how do I find out which script?

EDIT Correction, 354284 emails sent with 50 'to' addresses each.... 17,714,200 emails .... excellent.

EDIT Looks like an smtp user/bot net... the mails are being sent by an authenticated user....

Apr 22 06:31:41 impulsemedia relaylock: /var/qmail/bin/relaylock: mail from 71.129.165.22:25411 (adsl-71-129-165-22.dsl.irvnca.pacbell.net)
Apr 22 06:31:42 impulsemedia relaylock: /var/qmail/bin/relaylock: mail from 71.129.165.22:25412 (adsl-71-129-165-22.dsl.irvnca.pacbell.net)
Apr 22 06:31:42 impulsemedia relaylock: /var/qmail/bin/relaylock: mail from 71.129.165.22:25413 (adsl-71-129-165-22.dsl.irvnca.pacbell.net)
Apr 22 06:31:42 impulsemedia relaylock: /var/qmail/bin/relaylock: mail from 71.129.165.22:25414 (adsl-71-129-165-22.dsl.irvnca.pacbell.net)
Apr 22 06:31:42 impulsemedia relaylock: /var/qmail/bin/relaylock: mail from 71.129.165.22:25415 (adsl-71-129-165-22.dsl.irvnca.pacbell.net)
Apr 22 06:31:42 impulsemedia smtp_auth: SMTP connect from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:42 impulsemedia smtp_auth: smtp_auth: SMTP user --removed--@--removed--.com : /var/qmail/mailnames/--removed--.com/--removed-- logged in from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:42 impulsemedia relaylock: /var/qmail/bin/relaylock: mail from 71.129.165.22:25416 (adsl-71-129-165-22.dsl.irvnca.pacbell.net)
Apr 22 06:31:42 impulsemedia relaylock: /var/qmail/bin/relaylock: mail from 71.129.165.22:25417 (adsl-71-129-165-22.dsl.irvnca.pacbell.net)
Apr 22 06:31:43 impulsemedia smtp_auth: SMTP connect from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:43 impulsemedia smtp_auth: smtp_auth: SMTP user --removed--@--removed--.com : /var/qmail/mailnames/--removed--.com/--removed-- logged in from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:43 impulsemedia relaylock: /var/qmail/bin/relaylock: mail from 71.129.165.22:25418 (adsl-71-129-165-22.dsl.irvnca.pacbell.net)
Apr 22 06:31:43 impulsemedia relaylock: /var/qmail/bin/relaylock: mail from 71.129.165.22:25419 (adsl-71-129-165-22.dsl.irvnca.pacbell.net)
Apr 22 06:31:43 impulsemedia relaylock: /var/qmail/bin/relaylock: mail from 71.129.165.22:25420 (adsl-71-129-165-22.dsl.irvnca.pacbell.net)
Apr 22 06:31:43 impulsemedia smtp_auth: SMTP connect from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:43 impulsemedia smtp_auth: smtp_auth: SMTP user --removed--@--removed--.com : /var/qmail/mailnames/--removed--.com/--removed-- logged in from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:43 impulsemedia smtp_auth: SMTP connect from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:43 impulsemedia smtp_auth: smtp_auth: SMTP user --removed--@--removed--.com : /var/qmail/mailnames/--removed--.com/--removed-- logged in from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:43 impulsemedia smtp_auth: SMTP connect from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:43 impulsemedia smtp_auth: smtp_auth: SMTP user --removed--@--removed--.com : /var/qmail/mailnames/--removed--.com/--removed-- logged in from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:44 impulsemedia smtp_auth: SMTP connect from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:44 impulsemedia smtp_auth: smtp_auth: SMTP user --removed--@--removed--.com : /var/qmail/mailnames/--removed--.com/--removed-- logged in from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:44 impulsemedia relaylock: /var/qmail/bin/relaylock: mail from 71.129.165.22:25422 (adsl-71-129-165-22.dsl.irvnca.pacbell.net)
Apr 22 06:31:44 impulsemedia relaylock: /var/qmail/bin/relaylock: mail from 71.129.165.22:25421 (adsl-71-129-165-22.dsl.irvnca.pacbell.net)
Apr 22 06:31:44 impulsemedia smtp_auth: SMTP connect from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:44 impulsemedia smtp_auth: smtp_auth: SMTP user --removed--@--removed--.com : /var/qmail/mailnames/--removed--.com/--removed-- logged in from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:44 impulsemedia smtp_auth: SMTP connect from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:44 impulsemedia smtp_auth: smtp_auth: SMTP user --removed--@--removed--.com : /var/qmail/mailnames/--removed--.com/--removed-- logged in from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:44 impulsemedia smtp_auth: SMTP connect from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:44 impulsemedia smtp_auth: smtp_auth: SMTP user --removed--@--removed--.com : /var/qmail/mailnames/--removed--.com/--removed-- logged in from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:44 impulsemedia smtp_auth: SMTP connect from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:44 impulsemedia smtp_auth: smtp_auth: SMTP user --removed--@--removed--.com : /var/qmail/mailnames/--removed--.com/--removed-- logged in from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:44 impulsemedia relaylock: /var/qmail/bin/relaylock: mail from 71.129.165.22:25423 (adsl-71-129-165-22.dsl.irvnca.pacbell.net)
Apr 22 06:31:44 impulsemedia relaylock: /var/qmail/bin/relaylock: mail from 71.129.165.22:25424 (adsl-71-129-165-22.dsl.irvnca.pacbell.net)
Apr 22 06:31:44 impulsemedia relaylock: /var/qmail/bin/relaylock: mail from 71.129.165.22:25425 (adsl-71-129-165-22.dsl.irvnca.pacbell.net)
Apr 22 06:31:45 impulsemedia smtp_auth: SMTP connect from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:45 impulsemedia smtp_auth: smtp_auth: SMTP user --removed--@--removed--.com : /var/qmail/mailnames/--removed--.com/--removed-- logged in from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:45 impulsemedia smtp_auth: SMTP connect from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:45 impulsemedia smtp_auth: smtp_auth: SMTP user --removed--@--removed--.com : /var/qmail/mailnames/--removed--.com/--removed-- logged in from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:46 impulsemedia smtp_auth: SMTP connect from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:46 impulsemedia smtp_auth: smtp_auth: SMTP user --removed--@--removed--.com : /var/qmail/mailnames/--removed--.com/--removed-- logged in from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:46 impulsemedia smtp_auth: SMTP connect from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:46 impulsemedia smtp_auth: smtp_auth: SMTP user --removed--@--removed--.com : /var/qmail/mailnames/--removed--.com/--removed-- logged in from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:46 impulsemedia smtp_auth: SMTP connect from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:46 impulsemedia smtp_auth: smtp_auth: SMTP user --removed--@--removed--.com : /var/qmail/mailnames/--removed--.com/--removed-- logged in from unknown@adsl-71-129-165-22.dsl.irvnca.pacbell.net [71.129.165.22]
Apr 22 06:31:49 impulsemedia qmail-queue-handlers[28215]: Handlers Filter before-queue for qmail started ...
Apr 22 06:31:49 impulsemedia qmail-queue-handlers[28215]: from=mysteryshopping@stmarysalumni.com
Apr 22 06:31:49 impulsemedia qmail-queue-handlers[28215]: to=ctrudel0103@aol.com
Apr 22 06:31:49 impulsemedia qmail-queue-handlers[28215]: to=d__franco@msn.com
Apr 22 06:31:49 impulsemedia qmail-queue-handlers[28215]: to=d__john2258@yahoo.com
Apr 22 06:31:49 impulsemedia qmail-queue-handlers[28215]: to=d_1n_only@yahoo.com
Apr 22 06:31:49 impulsemedia qmail-queue-handlers[28215]: to=d_5boyz@yahoo.com
Apr 22 06:31:49 impulsemedia qmail-queue-handlers[28215]: to=d_85242@yahoo.com

Then 50 or more "to" addresses, reason I didn't catch this in the logs is they logged in here - dumped most of the emails in the queue then the rest of the 300m+ log is delivery messages looking like a script. That ip address '71.129.165.22' also shows up on the spamhause CBL ....

Just goes for a lesson to read my logs more carefully when there is a problem.

-sean

share|improve this question

6 Answers 6

up vote 0 down vote accepted

We got attacked by a spammer earlier this week as well. One piece of advice I found was to look at the full headers of the earliest spam message you can find and look for the invoked by UID. You can look this up in the password file to determine which login was used to run the process that sent the emails.

For what its worth, the entry point for the spam turned out to be our web mail interface. The spammer logged in using an existing account and password and then sent emails using the web mail application. From what I can tell, the spammer never compromised the actual system.

share|improve this answer
    
yea - what looks like happened was the user had their credentials stolen [somehow?] and a bot or something was using those to send the mail - I disabled the account and eveything stopped. The big clue was the IP address, the user is not far from me here [just outside of Toronto] but the originating IP is in California.... –  Sean Kimball Apr 25 '11 at 17:11

While you could just grep for 'mail(' that's not the only way to send an email from PHP. It could also be done via the various program execution functions (the standard mail fn is just a wrapper arounf the program defined in php.ini) or it might connect to a SMTP port.

Regardless, it should have taken some time to process that volume of messages - or required lots of HTTP requests - both of which would be evident in your webserver logs.

I would recommend replacing the command configured in the php.ini file with one which will gather as much info as possible about what has invoked it - and log it somewhere. Also, if you have got port 25 open on the machine, then block access to it from scripts (note that this may break the service you are providing to users).

share|improve this answer
    
I've been through the access/error logs on domains I consider 'suspicious' - nothing of note there. It did take a while to process, it appears to start at 6pm and go right through to about 4am, I'm really surprised I did not get any nagios alerts they were injected at a rate of between 4 and 10 emails per second, that should have tripped something. I was looking at the php.ini and thinking 'there must be a way to get more info' since it's just referencing /usr/bin/sendmail [actually it's qmail] .... –  Sean Kimball Apr 23 '11 at 23:09

I would look at apache access logs because there is a good possibility that there is a script in your webroot which utilizes the mail() function and it is not secure. I strongly suspect this to be a culprit. Quite possible the script is called mail.php

If that doesn't help then a way to brute force it would be to use grep, grepping the files for the mail() function.

share|improve this answer
    
well there's a couple hundred domains on the server, so I went about it looking for unusually large access, error, access_ssl & error_ssl log files .... archives too, nothing over a meg anywhere. if this was a script, it wasn't hit 83,000 times. [the mail log from last night is 340m gzipped!] –  Sean Kimball Apr 23 '11 at 21:29

If you are running a website, probably it's a hidden/unwanted feature in some webpage or a possible defacement. Look in the webserver access logs for similar accesses during the time gap that you think the mails were sent.

I'd also look for cronjobs that may be sending mail. If you are using Linux, type ls -la /var/spool/cron/crontabs/ to find all the users that have cron jobs installed and take a look at them.

Hope this helps!

share|improve this answer
    
hmmm I've checked [and tested!!] my relay settings, they are fine [!no relay!] found the sent messages in the logs, they are always: Apr 22 06:32:05 impulsemedia qmail-remote-handlers[3178]: from=mysteryshopping@stmarysalumni.com Apr 22 06:32:05 impulsemedia qmail-remote-handlers[3178]: to=d_lelievre@hotmail.com followed by 50 'to' addresses ..... –  Sean Kimball Apr 23 '11 at 21:48

You should create a wrapper that logs various information about the requests.

Parallels made an example for Plesk systems, but it looks somewhat generic: http://kb.parallels.com/1711

share|improve this answer
    
yes - it is, but it works, just a little tweaking and it should do just fine.... –  Sean Kimball Apr 23 '11 at 23:17
    
thanks FBH - I have that running now, I also found a way to use lsof to log these things,,, "lsof +r 1 -p ps axww | grep httpd | grep -v grep | awk ' { if(!str) { str=$1 } else { str=str","$1}}END{print str}' | grep vhosts | grep php > scripts.run &" takes a good chunk of CPU though... I guess now 'we wait' –  Sean Kimball Apr 23 '11 at 23:54

yea - what looks like happened was the user had their credentials stolen [somehow?] and a bot or something was using those to send the mail - I disabled the account and eveything stopped. The big clue was the IP address, the user is not far from me here [just outside of Toronto] but the originating IP is in California.... –

So thank you everyone for your input [all valid and useful techniques] but the problem was not an insecure script at all.

I did find the sendmail wrapper particularly useful. A trick I'll have to keep handy

-sean

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.