I have recently replaced a Fedora Core 3 server with one running CentOS 5 for the purposes of a mail relay in a DMZ.
My problem is that when the server is sending messages to a particular remote organizaion, messages larger than 500KB don't send properly. They seem to get hung up somewhere through the transmission, time out, then after the predictable retries, expire in the queue.
Sendmail describes the problem as: MDeferred: Connection timed out with [site].
Large (or larger) messages send successfully to every other remote system we have tested with. It is just this single organization which we are having trouble with. Similarly, we can send large or larger messages to this organization as long as our CentOS 5 relay is not involved.
We spent a lot of time with packet traces that were not very helpful. It appears that after a certain depth of transmission, the other side started requesting packet retransmission, which we did, but the retransmitted packets never seemed to reach their side.
Messing with iptables (ie turning it off completely) didn't help either.
Today we put an XP system in the DMZ as a peer to the relay, and it can send to the remote organization just fine while at the same time the relay could not. This, to my mind, rules out all the firewalls and network paths between us and the remote organization and points the finger directly at the mail relay.
Given that I am revisiting this sendmail after setting it up for Fedora Core 3, is there something wrong I might have done while setting this system that would manifest in this way?