1

When a spammer uses his botnet of thousands of zombie PCs to spam at random nonexistent addresses @example.com, at a very high rate of speed, postfix exhausts our VPS provider's resource limits trying to deal with it. Specifically, it exhausts the number of non-TCP sockets, which is limited to 900 by the VPS. Am running postfix 2.3.3-2.1.el5_2 on CentOS 5 on a Virtuozzo linux VPS.

/var/log/maillog says:
Feb 23 06:26:22 postfix/smtpd[3938]: warning: connect #1 to subsystem private/proxymap: Cannot allocate memory
Feb 23 06:26:22 postfix/smtpd[3936]: fatal: socket: Cannot allocate memory
Feb 23 06:26:48 postfix/qmgr[17702]: fatal: socket: Cannot allocate memory

Firewalling would be kind of tough because of the thousands of IPs involved in the dictionary attack.

The VPS provider suggested tuning the following parameters, but did not give suggestions on what to set them to:

max_idle = 100s (default)
max_use = 100 (default)

I found another person having the same problem with postfix and spammer dictionary attacks:

http://forums.vpslink.com/linux/394-you-hitting-socket-resource-limits-2.html#post5241

He changed:

default_process_limit from 100 (default) to 10

... which solved the problem but introduced a performance penalty.

I am unsure exactly which parameter should be safely tuned here, even after skimming postfix.org/TUNING%5FREADME.html Can any postfix expert help?

2 Answers 2

1

Sadly, as Postfix follows the process driven model, high memory usage under load is one of it's side effect. You could try this

From /etc/postfix/master.cf

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
smtp      inet  n       -       n       -       5       smtpd

In the maxproc column, you can replace the - with a smaller number to limit the number of concurrent smtpd processes, this should provide some gate on the amount of incoming mail.

Another alternative would be to look at fail2ban which can be configured to parse /var/log/maillog and raise iptables blocks for addresses that send a large amount of undeliverable mail.

5
  • Thank you Dave. Any suggestions on what the smaller number should be? This is a fairly active server with several hundred accounts and almost 60 different domains, so I'm not quite comfortable with only 5 processes yet. Right now, for example, there are 12 smtpd processes and no dictionary attacks going on (thankfully). On a similar server, there are 27 smtpd processes. Should I just double the higher number and make it say, 54 possible processes and fine-tune it based on trial & error? Or is there a more scientific way to do this?
    – ane
    Feb 23, 2010 at 21:09
  • Whoa, 12? How may cores do you have on this beast? What that means is you can have 12 simultaneous connections - which potentially means 12 very busy CPUs, if you have virus scanners attached to postfix. If you only have a quad or dual processor server, you'll want to trim that number up a bit, say 8 or 4. Yeah, you won't be able to get a connection while the spam storm is raging, but it will rate-limit the spammer too...and prevent your box from keeling over. I guess a better question is: how many users would realistically connect to send all at once? Take that number, and add 2. Feb 24, 2010 at 2:48
  • Ana - from your initial post it appeared your problem was memory usage, not cpu usage, because you were working in a small VPS. Don't forget that Postfix works on a per process model, which will fork one smtpd for the life of the smtp transation. That is from EHLO to closing the connection once the message has gone through your spam filters, etc. If you need more performance per mb of ram, Postfix probably isn't the server for you. Feb 24, 2010 at 5:02
  • The specific problem is the number of non-TCP sockets eaten by postfix during dictionary attacks (thousands) is more than the provider provides (max: 900). It appears that the more smtpd processes exist, the more non-TCP sockets are used (?) When postfix hits this limit, it gives a memory allocation error and stops accepting new connections. There is actually plenty of memory free (over 1GB) but the non-TCP sockets are maxed out.
    – ane
    Feb 24, 2010 at 16:18
  • Finally got around to trying Dave's advice. After decreasing it to 5, I now receive the following: Mar 26 12:00:03 postfix/master[23946]: warning: service "smtp" (25) has reached its process limit "5": new clients may experience noticeable delays Mar 26 12:00:03 postfix/master[23946]: warning: to avoid this condition, increase the process count in master.cf or reduce the service time per client So I set it back to the default. Guess some experimentation is needed to figure out the correct balance of max processes. Still not sure how maxproc affects non-TCP sockets.
    – ane
    Mar 26, 2010 at 16:04
0

You might want to consider adding something similar to:

smtpd_recipient_restrictions =
        permit_auth_destination,
        permit_mynetworks,
        reject_unauth_destination,
        reject_unlisted_recipient

...to your main.cf file, which should cause postfix to dump some of the connections as soon as it figures out there isn't a user to deliver to. The magic is in reject_unlisted_recipient, which will cause rejections for users that are not valid on your system when used with local_recipient_maps. By doing so, it should take some pressure off by reducing the amount of processing going on, as each dropped connection frees up precious resources to deal with the next connection. Yes, the spammer will be able to determine what addresses are invalid, but better that the spammer wastes their time sending (and being rejected, wasting their bandwidth) than yours (fending off the attack).

1
  • Thanks Avery. That's the default setting but I moved it higher in the smtpd_recipient_restrictions list. After monitoring, the number of non-TCP sockets remains about the same, so I'm not sure this particular setting has much effect on non-TCP sockets.
    – ane
    Feb 24, 2010 at 16:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .