Exchange OAB (OAL) will not generate after upgrade from Exchange 2007 to 2010 (8004010e)

Question

I've been pulling my hair out for days now, let me see if I can describe my problem.

Basically, we are 90% of the way through migrating to Exhcnage 2010 from 2007. Ever since I moved the OAB (or OAL now in 2010) to the new server, I have been getting these errors (this is with logging set to expert)

First I get the logs saying it is starting Then I get a whole bunch (which are aparently normal) the last being:

Log Name: Application 
Source: MSExchangeSA 
Date: 28/06/2010 8:46:42 a.m. 
Event ID: 9359 
Task Category: (13) 
Level: Warning 
Keywords: Classic 
User: N/A 
Computer:  
Description: 
OALGen truncated or dropped properties for entry 'Joe Tsang' in address list '\Global Address List' because they exceeded the configured size limits for the version 4 offline address list. The affected MAPI ids are: 8008. 
- \Default Offline Address Book 

I get about 60 or so of these, and then I get:

Log Name: Application 
Source: MSExchangeSA 
Date: 28/06/2010 8:46:42 a.m. 
Event ID: 9339 
Task Category: (13) 
Level: Error 
Keywords: Classic 
User: N/A 
Computer:  
Description: 
Active Directory PDC returned error 8004010e while generating the offline address book for '\Global Address List'. The last recipient returned by the Active Directory was 'Joe Tsang'. This offline address book will not be generated. 
- \Default Offline Address Book 

And:

Log Name: Application 
Source: MSExchangeSA 
Date: 28/06/2010 8:46:42 a.m. 
Event ID: 9330 
Task Category: (13) 
Level: Error 
Keywords: Classic 
User: N/A 
Computer:  
Description: 
OABGen encountered error 8004010e (internal ID 5001697) accessing Active Directory PDC for '\Global Address List'. 
- \Default Offline Address Book 

And finally:

Log Name: Application 
Source: MSExchangeSA 
Date: 28/06/2010 8:46:42 a.m. 
Event ID: 9126 
Task Category: (13) 
Level: Error 
Keywords: Classic 
User: N/A 
Computer:  
Description: 
OABGen encountered error 8004010e while calculating the offline address list for address list '\Global Address List'. This offline address list will not be available for client download. 
- \Default Offline Address Book. 

Also, I noticed during some checking that I am getting a bunch of errors in my OABInteg checking:

Processing Address Book Entry #23 of 50. 

Display Name = [email protected] 
Object is a Distribution list object 
ERROR: LegacyExchangeDN has the '@' character present. This is a PDN issue and this can cause full downloads. 

however the summary is:

Scan Finished 
+-------------+ 
Total number of entries processed in the address book: 258 
Total number of entries skipped: 0 
Total number of contacts: 4 
Total number of mailboxes: 139 
Total number of distribution lists: 83 
Total number of groups: 0 
Total number of folders: 32 
Total number of Address Book Container objects found: 0 
Total number of temp legacyExchangeDN's found: 0 
Total number of objects that are missing some main attributes: 0 
Total number of objects that mail and proxy attribute don't match: 0 
Total number of objects that do not have a domain value: 0 
Total number of objects that do not have a valid unicode domain value: 0 
Total number of objects that do not have a valid SMTP Domain because first character is not greater than '/': 0 
Total number of objects that do not start with /o= or /O=: 0 
Total user objects that are missing the Primary Proxy address attribute: 0 
Total user objects with proxy addresses equal to or over 64 characters: 0 
Total objects with the '@' character in the legacyExchangeDN: 8 
Total objects with bad Active Directory Backlinks: 0 
Total objects that have a legacyExchangeDN of ADCDisableMail: 0 
Total objects that have a legacyExchangeDN of ADCDisableMailByADC: 0 

Now I have spent a-l-o-t of time researching this, and it seems to be an issue going way back to Exchange 2003, and to do with Active Directory corrupt records (specifically the Manager, Secretary and HomeMDB and ClientMDB values). I ran a tool that was reccomended by Dave Goldman's blog that let me export the entire AD Database (and the correlating values that may be causing the issue) and after manually sifting through 14000 records I concluded that there are no malformed, null or corrupt values in any of those fields or that I lost my mind half way through and I missed them.

I have also tried:

1. Running OABInteg and the proxytest and alltests returns completely normal results.
2. Used the MAPI tool to open up my current OAB and manually checked the previous and following 50 records (as the OAB is generated in bunches of 50) and have found nothing.
3. I have completely remade the OAB (more than once) and get the exact same result. I have also tried using both the web and public folder distribution methods.

The only thing I can find that may help me in this is a mysterious tool developed by the Illumiati branch of Microsoft called "NSPITool" that apparently will in three easy steps tell me where the malformed values are in my active directory and let me resolve it. However this tool is aparently only available via PSS and i'd rather try post here first before contacting MSPSS.

Any help would be appreicated!!

Gareth

Answer 1

One step that helped me diagnose my own OAB/OAL problems post-2007-migration was to turn the OAB generation log level to high. That helped isolate some problem entries. I ended up having to re-set some values on a few mailboxes. Silly stuff like changing a specific SMTP address and changing it back to the original value, forcing a regeneration of the OAB, and seeing it work.

Changing the event log level: http://technet.microsoft.com/en-us/library/aa998905(EXCHG.80).aspx

Set-EventLogLevel -Identity “MSExchangeSA\OAL Generator” –Level Expert

That needs to be run on the server(s) hosting the Mailboxes. Not on the Hub or CA server(s).

Answer 2

MS Support PSS - oabvalidate.exe http://social.technet.microsoft.com/Forums/en-US/exchange2010/thread/6a5cebd7-4c64-4a00-98d8-f635c26c6c3e/