Erratic or Totally Missing eMails with Windows SMTP Server

This week I was doing configuration work on a client's new SharePoint Server 2010 farm to get inbound & outbound mail set-up.  This project had a slightly "one-off" requirement in that their business uses Microsoft Office365 for their employees' email and Office suite.  Ergo, there is no on-premise Exchange or other mail server and instead we needed to set-up SharePoint to route mail via their mail domain inside Office365.  Fortunately, John White has two great blog posts covering this topology (though it references Office365's "predecessor," BPOS):

http://whitepages.unlimitedviz.com/2010/09/integrating-sharepoint-on-premises-with-bpos-and-exchange-online-part-1outbound/

http://whitepages.unlimitedviz.com/2010/10/integrating-sharepoint-on-premises-with-bpos-and-exchange-online-part-2-inbound/

I had the configuration done and had started unit testing the setup.  Outbound email was working fine but as we completed more and more inbound email tests and confirmed the SharePoint site's mail-enabled document library was getting the emails it became apparent that not ALL emails were making it in.  Time to put on the troubleshooting faceshield, crap-waders, leather gloves and get out the proper tools for computer troubleshooting:

 

Spam filter folders/logs showed nothing amiss. Examination of the SMTP log files showed that each test message resulted in the Office365 mailservers connecting to our SMTP server and seemingly delivering the email messages successfully (e.g. 200-series status codes, bytes-transferred relative to the test email's attachments).  No error messages could be found in the SharePoint ULS nor the Windows Event Logs.  Then, looking in the SMTP server's Badmail folder (default location is C:\inetpub\mailroot\Badmail) we discovered where all the missing emails were going.

For each missing message sent, three files existed in the Badmail folder; a .bad, a .bdr, and a .bdp.

This MS KB article describes the three file types in more detail:  http://support.microsoft.com/kb/306043

But looking inside the .bdr files for an explanation gave only the following error information:

Unable to deliver this message because the follow error was encountered:This message is a delivery status notification that cannot be delivered.The specific error code was 0xC00402C7.

Sadly, that is a fairly generic error message and Googling it lead to many false leads (including the KB article above).


Over the course of several days we sent multiple messages from different email addresses (e.g. hotmail, gmail, internal to the client, etc.) and the puzzle only grew.  There seemed to be no absolute pattern to the failures. E.g. attachment v. no attachment, type of attachment, size of attachment, time of day, one sender v. another, one email service v. another, text in the body of the message.  Eventually, though, it started to seem that emails sent via corporate email servers (e.g. our Exchange server) had a higher failure rate.

So here is the short and skinny of it all:  The default setting in the MS SMTP Service (for IIS7.5...and probably all the way back to the first offering of SMTP Server) for the Maximum Hop Count is 15.

You can see that setting at the top of the Advanced Delivery tab as screenshotted in John White's coverage of SMTP configuration in his blog post (see URL above). Ignore his red circle around the Smart host parameter.....

In the old days, that was probably a pretty generous limit.  But, now days with outbound & inbound spam filters, corporate IP trackers, and global Web mail-provider networks (e.g. GMail), it seems it isn't that hard to have your email message passed through 15 machines on its zippity-do-da way around the big ole Internet as it goes to a server across the street from your office.  Depending on the gods-of-TCP/IP-routing and the current sum of the airspeed of all airborne English Sparrows, test messages that seemed to be identical tests resulted in different pass/fail results as some messages took slightly different (i.e. longer routes, they'd exceed the allowed Maximum Hop Count and the SMTP Server would do as configured and shove them into the Badmail folder.

 Raising the Maximum Hop Count to a more generous 99 (max allowed value is 255) solved the erratic-emails problem.

One other thing to note while your in the SMTP configuration is to ensure the message size and session limits on the Messages tab reflect modern email expectations...especially for a SharePoint server that will be accepting documents into a document library via email.  If nothing else, raise the "Limit Message Size To (KB)" parameter from its default value to something with real MB meat - e.g. 51200KB = 50MB (but don't forget that the default IIS & SharePoint file upload limits might still need to be addressed for your document library's needs, as well).

Sadly, the server never seemed to use the email address in the "Send Copy of Non-Delivery Report to:" parameter (aka Postmaster) to send an email notifying a human of the non-deliverable emails during our testing.

Oh, and here is a nice single-page overview sheet of the MS SMTP Server settings: http://www.tech-faq.com/understanding-and-managing-smtp-virtual-servers.html