5

Environment

  • Rackspace
  • Ubuntu 12.04
  • Wordpress
  • MySql

The issue

I have been experiencing quite serious out-memory-issues in the last couple of days.

While I resolved one possible cause the issue I still get a very suspicious activity of sendmail.

Any recommendations on how to tackle this issue? I think that must be some malware, but I no experience on resolving this kind of attacks.

htop

  1  [|||||||||||||||||||||||||                                                          27.0%]     Tasks: 101, 50 thr; 1 running
  2  [|||||||||||||||||||||||||||||||||||||||||                                          45.7%]     Load average: 12.96 12.55 11.95 
  Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||1183/1995MB]     Uptime: 09:53:28
  Swp[||||                                                                           93/2047MB]

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
19704 root       20   0  120M 25328  2896 S  2.0  1.2  0:46.16 sendmail: MTA: ./s6HH4rLv009027 gmail.co.: user open
 3298 root       20   0   99M  5612  1684 S  2.0  0.3  2:46.31 sendmail: MTA: s6OABpf4003298 localhost [127.0.0.1]: DATA
 3301 root       20   0   99M  5544  1684 S  2.0  0.3  2:40.89 sendmail: MTA: s6OAGAAh003301 localhost [127.0.0.1]: DATA
19510 root       20   0 26488  2568  1212 R  2.0  0.1  0:23.73 htop
  771 syslog     20   0  244M  3892   516 S  1.0  0.2  2:22.43 rsyslogd -c5
 1226 smmsp      20   0  133M 56328  1396 S  0.0  2.8  1:56.85 sendmail: MSP: ./s6K1OdvJ030780 [127.0.0.1]: client DATA status
32488 root       20   0  102M  7168  2748 S  0.0  0.4  0:00.02 sendmail: MTA: ./s6OAcr6I032488 aspmx.l.google.com.: client EHLO
31723 www-data   39  19  448M 72676 47276 S  0.0  3.6  0:01.14 /usr/sbin/apache2 -k start
29624 root       20   0  120M 25916  2884 S  0.0  1.3  0:29.65 sendmail: MTA: ./s6NHPdHs002287 todito.com.: user open
  898 mysql      20   0 1315M  105M  3296 S  0.0  5.3 23:25.23 /usr/sbin/mysqld
30966 root       20   0  101M  5092   460 D  0.0  0.2  0:01.52 sendmail: MTA: running queue: /var/spool/mqueue
 5013 mysql      20   0 1315M  105M  3296 S  0.0  5.3  0:25.58 /usr/sbin/mysqld
25504 root       20   0  120M 25904  2900 S  0.0  1.3  0:24.57 sendmail: MTA: ./s6JHcEdS028616 hotamil.com.: user open
 1033 root       20   0  630M  6228  2356 S  0.0  0.3  1:17.85 /usr/local/bin/driveclient --daemon
 1062 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.50 /usr/local/bin/driveclient --daemon
 1082 newrelic   20   0  107M  1576  1072 S  0.0  0.1  0:46.81 /usr/sbin/nrsysmond -c /etc/newrelic/nrsysmond.cfg -p /var/run/nrsysmond.pid
 1089 newrelic   20   0  107M  1576  1072 S  0.0  0.1  0:46.80 /usr/sbin/nrsysmond -c /etc/newrelic/nrsysmond.cfg -p /var/run/nrsysmond.pid
  822 syslog     20   0  244M  3892   516 S  0.0  0.2  1:35.12 rsyslogd -c5
 1061 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.80 /usr/local/bin/driveclient --daemon
 8532 root       20   0  105M  9444   460 D  0.0  0.5  0:06.40 sendmail: MTA: running queue: /var/spool/mqueue
31711 www-data   39  19  445M 75316 52764 S  0.0  3.7  0:01.50 /usr/sbin/apache2 -k start
27927 root       20   0  120M 25904  2900 S  0.0  1.3  0:32.35 sendmail: MTA: ./s6NKLEhE005721 yahoo.co.: user open
13821 mysql      20   0 1315M  105M  3296 S  0.0  5.3  2:25.39 /usr/sbin/mysqld
31924 mysql      20   0 1315M  105M  3296 S  0.0  5.3  0:49.12 /usr/sbin/mysqld
31713 www-data   39  19  446M 68484 45496 S  0.0  3.4  0:00.79 /usr/sbin/apache2 -k start
 4195 mysql      20   0 1315M  105M  3296 S  0.0  5.3  0:29.08 /usr/sbin/mysqld
 9799 mysql      20   0 1315M  105M  3296 S  0.0  5.3  2:29.95 /usr/sbin/mysqld
 2664 smmsp      20   0  133M 56424  1476 D  0.0  2.8  1:52.68 sendmail: MSP: ./s6K3MC7s027126 [127.0.0.1]: client DATA status
  853 syslog     20   0  244M  3892   516 S  0.0  0.2  0:47.23 rsyslogd -c5
31714 www-data   39  19  446M 68404 45420 S  0.0  3.3  0:00.73 /usr/sbin/apache2 -k start
31903 mysql      20   0 1315M  105M  3296 S  0.0  5.3  0:47.96 /usr/sbin/mysqld
 1063 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.40 /usr/local/bin/driveclient --daemon
31600 www-data   39  19  448M 71340 46228 S  0.0  3.5  0:00.92 /usr/sbin/apache2 -k start
 4308 mysql      20   0 1315M  105M  3296 S  0.0  5.3  0:28.28 /usr/sbin/mysqld
 1064 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.41 /usr/local/bin/driveclient --daemon
31727 www-data   39  19  447M 70324 45756 S  0.0  3.4  0:00.84 /usr/sbin/apache2 -k start
31725 www-data   39  19  447M 70340 45756 S  0.0  3.4  0:00.86 /usr/sbin/apache2 -k start
31724 www-data   39  19  447M 70548 45932 S  0.0  3.5  0:00.84 /usr/sbin/apache2 -k start
 1715 mysql      20   0 1315M  105M  3296 S  0.0  5.3  3:05.00 /usr/sbin/mysqld
23774 root       39  19  425M  6636  4676 S  0.0  0.3  0:06.00 /usr/sbin/apache2 -k start
 1065 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.35 /usr/local/bin/driveclient --daemon
 1060 root       20   0  630M  6228  2356 S  0.0  0.3  0:12.43 /usr/local/bin/driveclient --daemon
F1Help  F2Setup F3SearchF4FilterF5Tre

huge /var/mail

root@web:/var/mail# ls -alh
total 1.2G
drwxrwsrwt  2 root     mail 4.0K Jul 24 10:51 .
drwxr-xr-x 15 root     root 4.0K Jul 24 00:45 ..
-rw-rw----  1 munin    mail  83K Jul 19 18:48 munin
-rw-------  1 root     mail 1.1G Jul 24 10:51 root
-rw-rw----  1 www-data mail  98M Jul 23 22:34 www-data

My root mail account is continuously sending emails

NOTE: I have replace my domain by filtered.com

Return-Path: <MAILER-DAEMON>
Received: from localhost (localhost)
    by web.filtered.com (8.14.4/8.14.4/Debian-2ubuntu2) id s6OAqpuv010033;
    Thu, 24 Jul 2014 10:52:51 GMT
Date: Thu, 24 Jul 2014 10:52:51 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON>
Message-Id: <201407241052.s6OAqpuv010033@web.filtered.com>
To: <kara_velazquez@filtered.com>
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
    boundary="s6OAqpuv010033.1406199171/web.filtered.com"
Subject: Returned mail: see transcript for details
Auto-Submitted: auto-generated (failure)

--s6OAqpuw010033.1406199172/web.filtered.com--

From MAILER-DAEMON  Thu Jul 24 10:52:53 2014
Return-Path: <MAILER-DAEMON>
Received: from localhost (localhost)
    by web.filtered.com (8.14.4/8.14.4/Debian-2ubuntu2) id s6OAqq6J010047;
    Thu, 24 Jul 2014 10:52:53 GMT
Date: Thu, 24 Jul 2014 10:52:53 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON>
Message-Id: <201407241052.s6OAqq6J010047@web.filtered.com>
To: postmaster
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
    boundary="s6OAqq6J010047.1406199173/web.filtered.com"
Subject: Postmaster notify: see transcript for details
Auto-Submitted: auto-generated (postmaster-notification)

This is a MIME-encapsulated message

--s6OAqq6J010047.1406199173/web.filtered.com

The original message was received at Thu, 24 Jul 2014 10:52:52 GMT
from localhost
with id s6OAqq6I010047

   ----- The following addresses had permanent fatal errors -----
<audra_gray@filtered.com>
    (reason: 550-5.1.1 The email account that you tried to reach does not exist. Please try)

   ----- Transcript of session follows -----
... while talking to aspmx.l.google.com.:
>>> RCPT To:<audra_gray@filtered.com>
<<< 550-5.1.1 The email account that you tried to reach does not exist. Please try
<<< 550-5.1.1 double-checking the recipient's email address for typos or
<<< 550-5.1.1 unnecessary spaces. Learn more at
<<< 550 5.1.1 http://support.google.com/mail/bin/answer.py?answer=6596 sq8si14059110obc.83 - gsmtp
550 5.1.1 <audra_gray@filtered.com>... User unknown
>>> DATA
<<< 503 5.5.1 RCPT first. sq8si14059110obc.83 - gsmtp

--s6OAqq6J010047.1406199173/web.filtered.com
Content-Type: message/delivery-status

Reporting-MTA: dns; web.filtered.com
Received-From-MTA: DNS; localhost
Arrival-Date: Thu, 24 Jul 2014 10:52:52 GMT

Final-Recipient: RFC822; audra_gray@filtered.com
Action: failed
Status: 5.1.1
Remote-MTA: DNS; aspmx.l.google.com
Diagnostic-Code: SMTP; 550-5.1.1 The email account that you tried to reach does not exist. Please try
Last-Attempt-Date: Thu, 24 Jul 2014 10:52:53 GMT

--s6OAqq6J010047.1406199173/web.filtered.com
Content-Type: text/rfc822-headers

Return-Path: <MAILER-DAEMON>
Received: from localhost (localhost)
    by web.filtered.com (8.14.4/8.14.4/Debian-2ubuntu2) id s6OAqq6I010047;
    Thu, 24 Jul 2014 10:52:52 GMT
Date: Thu, 24 Jul 2014 10:52:52 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON>
Message-Id: <201407241052.s6OAqq6I010047@web.filtered.com>
To: <audra_gray@filtered.com>
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
    boundary="s6OAqq6I010047.1406199172/web.filtered.com"
Subject: Returned mail: see transcript for details
Auto-Submitted: auto-generated (failure)

--s6OAqq6J010047.1406199173/web.filtered.com--

ps -ef | grep sendmail

root@web:/var/mail# ps -ef | grep sendmail
smmsp     1226     1  0 00:45 ?        00:02:04 sendmail: MSP: ./s6KKDDVU014035 [127.0.0.1]: client DATA status
smmsp     2644  2641  0 01:00 ?        00:00:00 /bin/sh -c test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp
smmsp     2647  2644  0 01:00 ?        00:00:00 /bin/sh /usr/share/sendmail/sendmail cron-msp
smmsp     2664  2647  0 01:00 ?        00:01:58 sendmail: MSP: [127.0.0.1]: idle              
root      3298     1  1 07:57 ?        00:03:16 sendmail: MTA: s6OB1dam003298 localhost [127.0.0.1]: DATA
root      3301     1  1 07:57 ?        00:03:05 sendmail: MTA: server localhost [127.0.0.1] cmd read
root     19675     1  0 11:20 ?        00:00:00 sendmail: MTA: ./s6OBKJuv019675 aspmx.l.google.com.: client DATA 354
root     19689     1  0 11:20 ?        00:00:00 sendmail: MTA: ./s6OBKLuv019689 aspmx.l.google.com.: client DATA 354
root     19800     1  0 11:20 ?        00:00:00 sendmail: MTA: ./s6OBKbuv019800 aspmx.l.google.com.: client DATA 354
root     20178     1  0 11:21 ?        00:00:00 sendmail: MTA: ./s6OBLSuv020178 aspmx.l.google.com.: client DATA 354
root     20270     1  0 11:21 ?        00:00:00 sendmail: MTA: ./s6OBLZuv020270 aspmx.l.google.com.: client DATA 354
root     20537     1  0 11:21 ?        00:00:00 sendmail: MTA: ./s6OBM0uv020537 aspmx.l.google.com.: client DATA 354
root     20646     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBM5uv020646 aspmx.l.google.com.: client DATA 354
root     21006     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMZ6I021006 aspmx.l.google.com.: client DATA 354
root     21015     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMZ6I021015 aspmx.l.google.com.: client DATA 354
root     21027     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMauv021027 aspmx.l.google.com.: client DATA 354
root     21036     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMb6I021036 aspmx.l.google.com.: client DATA 354
root     21063     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMeuv021063 aspmx.l.google.com.: client DATA 354
root     21065     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021065 aspmx.l.google.com.: client DATA 354
root     21086     1  1 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021086 aspmx.l.google.com.: client DATA 354
root     21094     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021094 aspmx.l.google.com.: client DATA 354
root     21098     1  2 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021098 aspmx.l.google.com.: client DATA 354
root     21103     1  1 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021103 aspmx.l.google.com.: client DATA 354
root     21105     1  1 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMguv021105 aspmx.l.google.com.: client DATA 354
root     21108     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OB1dag003298 mx-eu.mail.am0.yahoodns.net.: client MAIL
root     21111     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OBMg6I021111 aspmx.l.google.com.: client RCPT
root     21113     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OAsOi1003301 mx-ha03.web.de.: client greeting
root     21117     1  1 11:22 ?        00:00:00 sendmail: MTA: ./s6OAsOi3003301 gmail-smtp-in.l.google.com.: client DATA status
root     21123     1  0 11:22 ?        00:00:00 sendmail: MTA: ./s6OAsOi5003301 gmail-smtp-in.l.google.com.: client EHLO
root     21127 18604  0 11:22 pts/0    00:00:00 grep --color=auto sendmail

Sendmail status

root@web:/var/mail# /etc/init.d/sendmail status
MSP: is run via cron (20m)
MTA: is not running
QUE: Same as MTA

/var/spool/mqueue

root@web:/var/spool# ls -alh
total 48M
drwxr-xr-x  7 root  root  4.0K Mar 29  2013 .
drwxr-xr-x 15 root  root  4.0K Jul 24 00:45 ..
drwxr-xr-x  5 root  root  4.0K May  1  2012 cron
lrwxrwxrwx  1 root  root     7 May  1  2012 mail -> ../mail
drwxr-s---  2 smmta smmsp  14M Jul 24 11:44 mqueue
drwxrws---  2 smmsp smmsp  34M Jul 24 12:25 mqueue-client
drwxr-xr-x  2 root  root  4.0K Apr 13  2012 plymouth
drwxr-xr-x  2 root  root  4.0K Mar 30  2012 rsyslog


root@web:/var/spool# du -h -d 1
4.0K    ./plymouth
1.6G    ./mqueue    <=====
4.0K    ./rsyslog
One message from /var/spool/mqueue
root@web:/var/spool/mqueue# more qfs6OBTUZY003298 
V8
T1406201622
K1406201622
N1
P120781
I202/1/476577
MDeferred: 421 4.7.1 : (DYN:T1) http://postmaster.info.aol.com/errors/421dynt1.html
Fbs
$_localhost [127.0.0.1]
$rESMTP
$sweb.anybots.com
${daemon_flags}
${if_addr}127.0.0.1
S<patty_jennings@anybots.com>
MDeferred: 421 4.7.1 : (DYN:T1) http://postmaster.info.aol.com/errors/421dynt1.html
rRFC822; neve7@aim.com
RPFD:<neve7@aim.com>
H?P?Return-Path: <?g>
H??Received: from web.anybots.com (localhost [127.0.0.1])
    by web.anybots.com (8.14.4/8.14.4/Debian-2ubuntu2) with ESMTP id s6OBTUZY003298
    for <neve7@aim.com>; Thu, 24 Jul 2014 11:33:42 GMT
H??Received: (from www-data@localhost)
    by web.anybots.com (8.14.4/8.14.4/Submit) id s6JHVJId026134;
    Sat, 19 Jul 2014 17:31:19 GMT
H??Date: Sat, 19 Jul 2014 17:31:19 GMT
H??Message-Id: <201407191731.s6JHVJId026134@web.anybots.com>
H??X-Authentication-Warning: web.anybots.com: www-data set sender to patty_jennings@anybots.com using -f
H??To: neve7@aim.com
H??Subject: Fw:  Hi Generic Drugs Online Products
H??X-PHP-Originating-Script: 33:dirs.php
H??From: "Patty Jennings" <patty_jennings@anybots.com>
H??Reply-To:"Patty Jennings" <patty_jennings@anybots.com>
H??X-Priority: 3 (Normal)
H??MIME-Version: 1.0
H??Content-Type: text/html; charset="iso-8859-1"
H??Content-Transfer-Encoding: 8bit
.
  • 1
    What's in your mail logs? – MadHatter supports Monica Jul 24 '14 at 10:47
  • I updated my question accordingly. Looks like my root account is continuously send emails out. I can I stop this? – zabumba Jul 24 '14 at 10:56
  • 1
    Could we get the whole of one message from root's mailspool, rather than bits of two? Each new message will start with From (note trailing space) on a new line. It would also be helpful if you didn't redact your domain. – MadHatter supports Monica Jul 24 '14 at 10:58
  • I figured that I could just enter mail and get a sent email, but the server is getting very slow and I am not able to start mail. Do you have a hack to suggest? BTW I am filtering the domain cause I do have enough trouble. Don't want to expose my domain unnecessarily. I suppose you will understand. I wouldn't mind sharing with you privately. – zabumba Jul 24 '14 at 11:11
  • 2
    Stop sendmail so that the server isn't dying under load while you look into the problem. I understand the desire to redact the domain, but you're having the problem already; surely anything that helps us fix it is useful. I can point you at any number of questions on SF that were only solved because questioners didn't redact details. – MadHatter supports Monica Jul 24 '14 at 11:16
5

Your problem may be caused by HUGE number of (spam) messages in both sendmail queues.
(see https://serverfault.com/a/490890/163277 )

Check number of message in both sendmail queues

sendmail -O QueueSortOrder=none -Am -bp
sendmail -O QueueSortOrder=none -Ac -bp 

The most memory consuming sendmail process looks like MTA queue processing (-Am). The remaning looks like transfers from MSA to MTA queue and first time delivery attempts to external servers after such transfer.


You may use qtool.pl script to move messages send by www-data (web server) to another queue/directory. It is provided in contrib directory of sendmail.org distribution and in sendmail-base package by Debian-Linux.

  • Yes indeed, that looks like it. The first command you propose is spitting out hundreds of deferred emails. How do I empty these queues? and what could be causing this? – zabumba Jul 24 '14 at 12:03
  • Ok I found some suggestion here ... I will update my question accordingly. I still need to understand how this happened though. Puzzling me – zabumba Jul 24 '14 at 12:12
  • 2
    The appearance of user www-data in the logs suggests that you may have a web form that's allowing people to send arbitrary content to arbitrary recipients. Do the dates on these messages correlate with anything in your web logs? – MadHatter supports Monica Jul 24 '14 at 12:30
  • 1
    @MadHatter AFAIK you need logging at web server level. Sendmail sees only sending user (one www-data rubber bag). Wordpress seems to use PHPMailer class - the following link MAY be helpfull [Iresult of very quick search] howtoforge.com/… – AnFi Jul 24 '14 at 14:51
  • Guys, both your input and help has been huge. I wouldn't have nailed it without you. The best help in 4 years using the StackExchange. We had web forms injected into our Wordpress before, so it's a great lead. I have made an rsync copy of the whole folder before I started fixing up my instance. I am currently investigating the cause. I will keep you posted on my findings. – zabumba Jul 24 '14 at 21:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.