Limiting CPU/IO usage for linux services

Question

I've seen EXIM crash a system when it gets loaded by 10000s of e-mails from a user/script. I was wondering if there was a way to limit it's usage on a system and protect the system or service from abuse.

Answer 1

There are a few things that exim can do to reduce load.

1. Detect high load and make exim queue the large number of incoming messages generated by the script instead of trying to deliver it. The following options are relevant for that:

   queue_only_load         no immediate delivery if load is high
   queue_only_load_latch   don’t re-evaluate load for each message
   queue_only_override     allow command line to override
   

2. You're likely used to grepping or using exigrep to extract information from your /var/log/exim/main.log file (or whatever it's named for your distro). Exim also by default maintains a per message copy of this log, grouping each message's mail logs in a single file, deleting it when the message is delivered. You can view these for a single message with 'exim -Mvl QUEUEID'. Or just turn it off:

   On a heavily loaded system, it may be desirable to disable the use of per-message logs, in order to reduce disk I/O. This can be done by setting the message_logs option false.

3. Test using syslog to log to a remote server instead of locally. This will slightly reduce the disk I/O, though I think it will pale in comparison to the I/O caused by the large volume of email creation and such.
4. When you are done injecting emails into the mail server, load will drop. At that point, start up a bunch of queue runners. Exim is known for being slow to deliver messages from the queue, but if you have low load there is nothing wrong with starting up enough delivery processes to run the queue and deliver the mail quicker...until the point that your load becomes inhibitive (typically indicates that disk I/O utilization is reaching saturation). Try creating 10 queue runners at a time. I suspect you could probably do 100 or 150 without an excessive load. 'iostat -x -d 1' is your friend to make sure disk I/O is or is not a bottleneck.
5. Verify DNS both forward and reverse for your outbound mail server. Verify that the IP that recipient servers see is the same as you think it is and THAT must also be forward and reverse resolvable, else you will have slowdowns as the recipients do dns lookups on your IP (for their logs and such).

Answer 2

I suggest you check tools 'nice' and 'ionice' one to reduce impact of demanding process on CPU usage and other to reduce impact on IO tasks. (CFQ scheduler needs to be used if you wan't to use ionice for disk throughput throttling)

Alternatively you can check kernel cgroups subsystem. Cgroups support creating hard and soft memory limits as well as throttling of disk usage. (CFQ scheduler needs to be used for disk usage throttling)

Answer 3

You might be able to do what you need with PAM Limits (/etc/security/limits.conf) or ulimit.