0

My system sends email from php using sendmail. I realize that there are dozens of questions on stack exchange about delays sending email from sendmail/php. I have read them all but I still can't solve my problem.

I have a production server and a development server. Production is in a data center and uses a 'smart host' to relay its emails, direct sending of emails is blocked by a firewall. Development is in our office and sends emails directly. They also use different DNS servers. Other than this, these servers are as similar as I could possibly make them. My system needs to send many (around 10,000) emails at a time. My development server is able to send all emails in well under an hour. The production system takes more than 24 hours. The emails are 2kb-5kb in size.

On my production server:

I have determined that the hand-off from php to sendmail is fast. Sendmail is slow at sending emails to some domains. The big email services (gmail, aol, yahoo) are fast. I have tried to send 1000 emails to only aol addresses and they all sent within a couple minutes, so I know the relay is capable of sending emails quickly. My problem is that most of the email addresses in my address list are to small domains, and most of those are slow. This leads me to believe the slow step involves DNS. Here is an excerpt of my maillog file (values in {curly brackets} replaced for privacy).

Mar 28 11:21:47 {servernamereplaced} sendmail[26242]: v2SILe8w026242: to={usernamereplaced@domain1replaced.net}, ctladdr={customfromaddress@notmydomain.com} (48/48), delay=00:00:07, xdelay=00:00:00, mailer=relay, pri=32561, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (v2SILlIu026244 Message accepted for delivery) 
Mar 28 11:22:08 {servernamereplaced} sendmail[26247]: v2SILmT8026247: to={usernamereplaced@domain2replaced.net}, ctladdr={customfromaddress@notmydomain.com} (48/48), delay=00:00:20, xdelay=00:00:10, mailer=relay, pri=32550, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (v2SILwOh026248 Message accepted for delivery).

You can see there are long delays on these. However, when I run host -t mx domain1replaced.net I get a result very fast (less than half a second, and exactly as long as gmail.com or aol.com).

I've seen some answers mentioning the timeout values in sendmail.cf, however I don't know enough about that file to mess around on my own. Also, the delay values aren't all the same or multiples of the same number. They seem to be pretty scattered between 5 and 30 seconds. The thing that gets me is that my development server sends them so fast, but I can't get my production server to do the same thing. What can I do to speed up my production server?

  • 1) Sendmail asks DNS not only about MX records 2) your "followup DNS query" may access data stored in local DNS caching server - the first/initial query may be slower – AnFi Apr 1 '17 at 0:24
  • Well, email throttling setupped??? as if you send too fast you can be blacklisted – yagmoth555 Apr 1 '17 at 15:20
  • enable tls, that way AOL is equally slow. – Jacob Evans Apr 2 '17 at 15:58
  • I was able to speed up sending. I changed the way the production server communicates with the smart host. Instead of having php use Sendmail to send the email to the smart host I am now having php send with smtp to my smart host. I am now sending at about 100-150 emails per minute which is acceptable. – Owen Apr 5 '17 at 16:04
1

I am going to assume if you send another email to an address that was previously slow it will still be slow the second time. If that is the case I would try using telnet on your mail server to connect to the remote mail server and send a test email this way. I have weeded out a few delay related issues in the past with this method. It can sometimes help pinpoint what step in the communication process is holding things up.

Decent writeup on mailing with telnet:

https://weblogs.asp.net/owscott/Troubleshooting-email_2C00_-the-Telnet-way

  • Since my production server uses a smart host, would telnet-ing from the production server be of any use? Or would I need to test telnet from the smart host? – Owen Apr 3 '17 at 16:07
  • I would test from the prod server to the smart host, and then again from the smart host to the recipient domains mail server. This means you should also verify the rdns and SPF of both the prod server and the smart host, unless they are in the same private subnet, then that will only apply to the smart host. – alaphoid Apr 3 '17 at 17:58
1

The problem may not be with you locally doing DNS queries, but remote ends. When your SMTP server A sends to SMTP server B, server B will (may) do a PTR query on A IP address, and then a forward query on the name returned to make sure they match. If any of these DNS queries is slow, it may explain your observations.

1

Lets start analyzing sendmail's log, especially delay and xdelay. From here:

delay: The total message delay: the time difference between reception and final delivery or bounce). Format is delay=HH:MM::SS for a delay of less than one day and delay=days+HH:MM::SS otherwise

xdelay: The total time the message took to be transmitted during final delivery. This differs from the delay= equate, in that the xdelay= equate only counts the time in the actual final delivery.

To better emphasize the difference between delay/xdelay (taken from here):

This differs from delay= in that delay= shows the total amount of time the message took, computed from when the message was originally received or queued (this could be days ago), until it was eventually delivered. In the case of SMTP mail, the xdelay= computation starts when sendmail starts trying to connect to the remote host.

Your logs shows the following delay/xdelay:

delay=00:00:07, xdelay=00:00:00: this message waited for 7 seconds in sendmail's queue before smtp transfer was even being processed. This is a sign of an overloaded server, rather than connection/DNS problems: after all, the actual trasfer was executed almost instantly;

delay=00:00:20, xdelay=00:00:10: this time, the message stalled in sendmailìs queue for 10 seconds, and the actual transfer tool another 10 seconds. Based on your server's reputation, this can be a very reasonable value: many mailserver delay HELO/EHLO by 5-30 seconds for unknow of neutral senders.

So, why the test machine is so much faster than your production machine? Here are some possibilities:

  • sending through a smart host is much simpler and faster than sending email directly. When sending from your dev server, are you sure than email are actually received in about one hour, or this is only the time your dev server take to pass all mails to the smarthost?
  • maybe your smarthost has a good reputation and remote servers take its mails without forcing artificial waiting;
  • sending many emails is a very fsync() intensive work. Are your dev and production servers really the same, hardware and software wide? What about the smarthost's hardware and software stack?
0

Sending email to 100 domains will always be slower than sending email to 100 users at a single domain due to connection reuse, (postfix keeps log of the number of connection used) There's a ton of work involved in warming IPs and optimizing send rates per domain. If you add tls to the mix you'll also loose connection reuse options as each email builds a connection (tcp+tls builds plus recipient rate limits)

if you don't have or want to spend the time on building up reputation and tuning rate limits, you should evaluate 3rd party options.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.