Let me start by 'fessing up: I'm a programmer, not a system admin. (Although I used to be, over a decade ago. Let's be honest: That doesn't count anymore!)
We had a situation with a client last night where three e-mails, which are part of a recurring process and sent to the client, myself and the client's customers, were sent "inadvertently." Apparently they'd either been queued or in badmail, and got resent after a server bounce due to a software upgrade.
I cannot tell you if they were in queue, badmail, drop, etc. I didn't find out until after the damage was done. But all three e-mails were old enough to cause my client customer service issues: One was from October, 2008, and the other two from earlier this month. The client's customer service reps -- and I, as well -- received these e-mails on the original send dates. We do not know if the client's customers did or not.
What would be some good troubleshooting tips to start diagnosing this, and to help ensure that it doesn't happen again?
A bit about the machine in question:
- It's a shared web server, hosting applications and sites for dozens of clients.
- All of these clients go through the same SMTP service, which is provided by IIS 6. (It's a Windows 2003 machine.)
- This is the first time (as far as we know) that something like this has ever happened.
- These e-mails are generated as part of a scheduled, recurring process which executes at 6 AM each morning. (Indeed, the first thing I had to do upon waking up and seeing these on my phone was VPN in and check to make sure that the process didn't inadvertently execute. Luckily, it didn't, and I realized the age of these e-mails.)
- They were sent immediately (or very shortly) after two server bounces due to software upgrades. (One bounce was a Windows Update, the other was .NET 3.5 SP1.)
- The expiration timeout for the SMTP server is set 2 days. The Delay notification is set to 12 hours.
Finally, I'm not terribly well-versed in reading the SMTP logs. (I can read the W3SVC logs no problem, but the SMTP ones I'm not familiar with the numeric status codes.) So any help there would be appreciated as well.