1

I noticed the following blurb in the batbook:

The best way to flush a full queue is with a command line something like this:

/usr/sbin/sendmail -OQueueSortOrder=filename -q10m -d99.100

/usr/sbin/sendmail -OQueueSortOrder=random -q10m -d99.100 ← V8.12 and later /usr/sbin/sendmail -OQueueSortOrder=none -q10m -d99.100 ← V8.13 and later

Here, the -d99.100 tells sendmail to run in the foreground (so that you can kill it easily when done). The -q10m causes a queue-processing daemon to be launched once each 10 minutes (just like before). You need this because one daemon can seem to hang when delivering mail to a slow host. By running parallel daemons, you avoid this pitfall.

Later on in the same chapter, it talks about setting the interval to be a bad idea:

An hour later, service is restored. First, the default:

/usr/sbin/sendmail -q10m causes a forked copy of sendmail to start processing the queue. This time, however, the processing is not swift. When a queue fills to 30,000 or more messages, the amount of time it takes to preread the queue (to open and read every message) increases to more than 20 minutes.[180] And those 20 minutes are only for the preread. During those 20 minutes no mail will be sent.

After that, things get worse. Ten minutes later a second sendmail daemon is forked, and it, too, starts to preread the queue. Now, instead of one sendmail daemon opening and reading all messages in a queue, we have two sendmail daemons doing the same thing in parallel.

Contrary to what you might think, twice as much I/O on a disk is not twice as fast. Disks are finite devices that perform a limited number of disk-head moves[181] per second and can transmit only a fixed number of bytes per second. Because the two sendmail daemons are 10 minutes out of step with each other, each is reading and processing separate files. Depending on the size of your in-memory disk cache, neither will likely be able to take advantage of the efficiencies of such caching. In short, two sendmail daemons processing a deep queue in parallel is worse than a single sendmail daemon processing that same queue alone.

And if that weren’t enough, another 10 minutes later a third sendmail daemon starts to process the queue.

By now, the first sendmail daemon might have finished its preread of the queue and might have actually begun to send messages. But even if it has, three sendmail daemons are now processing that single deep queue and a curious thing happens. Because the disk that holds the queue is finite, the addition of a third sendmail daemon slows the operation of the first two. The second one, instead of taking 20 minutes to preread the queue, will now take 30 minutes.

This means that every 10 minutes another sendmail queue-processing daemon is added to the mix. As each is added, each slows all the others that are already running, and it isn’t long before the load on the machine starts to climb and the rate at which messages are delivered falls at an alarming rate. In fact, when this sort of behavior hits a very large-volume site, a sendmail queue-processing daemon can start and seem to never finish.

So I feel like I'm reading two completely different things in the documentation:

  • "hey! If your queue is massive, set the queue runner to launch every 10 minutes!"
  • "Hey! If your queue is massive, whatever you do, don't set the queue runner to be too short or it will slow stuff down."

What's the appropriate action to take for large queues?

2
  • Do you want the risk assessment? Do you want a fix?
    – AnFi
    Jun 16, 2017 at 2:46
  • @AndrzejA.Filip Sorry - I think my question was unclear. I feel like I'm seeing inconsistent recommendations on whether to run queue runners in parallel and would like clarification.
    – Mike B
    Jun 16, 2017 at 4:38

1 Answer 1

1

What you see is typical sendmal-ish "fix only one problem, ignore other possible problems" - it is caused by decades of "I contribute to fix MY problem" development.
The first advice deals with "slow to process" items in pretty big (but not huge) queue.
The second advice deals with HUGE queue processing which makes default sendmail queue run setup incredibly slow and memory hungry.

The first advice may be improved by using
* MinQueueAge - time between delivery attempts of specific message.
* MaxQueueRunSize- maximum number of messages processed by queue runner.
Old RFC recommends 30m delay between delivery attempts. Other MTA can used different delays between delivery attempts based on time spend in queue.
BTW Processing of MSA queue(s) and main queue(s) SHOULD be optimized separately.

IMHO: Most non big sites can deal with HUGE queue manually. It is most frequently caused "once per a few years" by a spam run due to hacking/vulnerability. It does not make sense to optimize sendmail to send it faster. See 18GB in MSA queue report.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .