2

I'm in the middle of migrating from Exchange 2013 to 2016. Most of the moves have gone well, however, some mailboxes seem to be getting stuck at Initial Seeding, and then after some hours of stalling here, fail at FailedStuck.

Running Get-MailboxStatistics -IncludeReport shows that the move is stuck running IsInteg because they are large mailboxes. This seems to basically be a New-MailboxRepairRequest that just gets queued indefinitely.

I can run New-MailboxRepairRequest with -Force and it will complete immediately, so I think it's just a workload management issue, but it's been a few days and repairs are still not completing.

Possible Solutions

  • How can I tell New-MoveRequest to skip the IsInteg run upfront? I will happily manually run them after the move.

  • How can I edit the Mailbox Repair Requests that the Move Requests creates, with the -Force flag? As far as I can tell there is no Set-MailboxRepairRequest.

  • How can I disable workload management for mailbox repair requests? As this seems to be what is stalling my moves/repairs, even when nothing is going on (it's the weekend at 4 AM on new hardware, and they won't budge).

  • How can I raise the "Large Mailbox" threshold (message below alludes to it being a config option). If I set it to something like 50GB, all of the mailboxes will be moved without doing a repair.

More Info

The report shows:

Report : 18/03/2017 12:34:12 AM [EXCH16-SYD-01] Setting up ISInteg repair run upfront for this mailbox since it's a large mailbox. 
"Primary mailbox size = 11120110046, Archive mailbox size = 0, Large mailbox size threshold config value = 10737418240

Then:

18/03/2017 12:54:33 AM [EXCH16-SYD-01] Store IsInteg task is pending completion for mailbox '6e6a0983-02ab-4d1d-84ea-d0071e7e6536'. IsInteg

The report repeats this for hours, until it eventually times out because no progress has been made. I eventually found that there is a corresponding

To prove that the workload management is causing problems, I checked for all mailbox repair requests across all mailboxes, and confirmed none of them were in progress.

[PS] C:\Windows\system32>Get-Mailbox -Server EXCH01-SYD | Foreach { Get-MailboxRepairRequest -Mailbox $_.PrimarySMTPAddress.tostring() }

Identity Task Detect Only Job State Progress
-------- ---- ----------- --------- --------
62d54094-6b61-4f1c-a... {MessageId} False Queued 0
...
62d54094-6b61-4f1c-a... {FolderView} False Queued 0

There's 29 repairs here, all of them are queued.

Things to Note

  • This only happens to mailboxes 10+ GB in size, because that's the threshold limit that causes an IsInteg task to be run before doing the move.

  • It's the IsInteg that is causing things to get held up. Yet if I do an IsInteg repair with -Force then the repair happens immediately.

  • Even for the IsInteg repairs I am able to do with -Force (which show as successful), I do not see any event log entries like I am supposed to.

  • My environment is 1x 2013 Exchange and 2x 2016 Exchanges, and 1x 2010 Exchange. All are running the latest CU or RU version.

  • Moving from 2010 is not a problem, even for mailboxes > 10 GB.

  • I've been running in Co-existence for a few weeks and migrated my own mailbox from 2013 to 2016 as a test. My mailbox is 15 GB and it did not get queued. This leads me to think if I was patient enough, the others might succeed. But my mailbox did not show FailedStuck and showed constant progress.

  • I am wondering if I just need to wipe out all the failed moves (right now I simply resume them) and re-try them one by one.

I feel like I've exhausted most of my options, so any advice is appreciated. Even if someone could tell me how to execute some of the "possible solutions" listed. As these are just theoretical solutions to me, I haven't found ways to actually do them.

  • Its not specific to the error you see, but in my migration I always had trouble in the new server's log, I usually enable circular logging for the importation. I give the tip as it seem to fail on big mailbox, its maybe not related at all, but in case it help you can try it – yagmoth555 Mar 19 '17 at 17:56
  • @MagicallyDelicous Hah, that's a great idea. Way to think outside the box. Will definitely be trying that. Large attachments are a definite possibility, but if that was the issue I'm not sure how I'd work around it. – Geekman Mar 20 '17 at 0:54
  • @MagicallyDelicous Unfortunately most of the bigger mailboxes are exec level people at the respective customers. Unfortunately with the recent addition of "Archive mailboxes" the question is often asked "Can't it just go in my archive mailbox?" :P – Geekman Mar 20 '17 at 1:06
  • @MagicallyDelicous Damn. So I just tried this and it's still doing the same thing even when attempting to move to 2010. So I guess that at least confirms that it's my 2013 system that's trying to do the repair, and causing the issue. It's got nothing to do with 2016. – Geekman Mar 20 '17 at 1:30
  • @MagicallyDelicous Still trying to figure it out. Windows updates not applicable, as all boxes are newly built/patched (except 2010) and all boxes have latest CU (including 2010). So no luck there. Going to try rebooting 2013 box over the weekend, but it's the only node left in a 2 node DAG, so not sure what surprises will appear. – Geekman Mar 21 '17 at 11:15
1

I found a simple solution to this problem: Downgrade your Exchange 2013 box.

As part of my migration plan, we had 2013 also in its own 2-node DAG. The 2nd node was brand new and had been patched up to the latest CU at the time:

AdminDisplayVersion : Version 15.0 (Build 1236.3) (Exchange 2013 CU14)

The old 2013 node was just patched enough for 2016 co-existence:

AdminDisplayVersion : Version 15.0 (Build 1178.4) (Exchange 2013 CU12)

I recalled that before I'd built the new 2013 box, I was successfully able to migrate a few 15 GB mailboxes over to 2016 without any issues.

During the course of my migrations, I had activated all the mailbox databases on the new node, so of course the CU version on the new node meant all future mailbox moves required a mailbox repair when > 10GB.

The difference is: for the older CU patch level, no mailbox repair is required over 10GB. Similar to Exchange 2010 in this sense. So because no repair/isinteg took place, the move succeeds. I am not sure exactly at what point Exchange 2013 began to require an IsInteg check as part of the migration. I am also not sure if this is "fixed" in later versions.

If need to downgrade to a lower CU, you might be able to build a new 2013 box, setup a DAG and then migrate all your databases over to the older versions.

You can see by comparing the move request reports, that IsInteg is run before any attempt is made to begin transferring the mailbox data, and this is why everything stalls.

Here's an excerpt of a move request that failed, due to the IsInteg check:

    18/03/2017 12:34:12 AM [EXCH16-SYD-01] Setting up ISInteg repair run upfront for this mailbox since it's a large mailbox. "Primary mailbox size = 11120110046, Archive mailbox size = 0, Large mailbox size threshold config value = 10737418240

    ...

    18/03/2017 12:52:58 AM [EXCH16-SYD-01] Connected to source mailbox '6e6a0983-02ab-4d1d-84ea-d0071e7e6536 (Primary)', database
    18/03/2017 12:52:58 AM [EXCH16-SYD-01] Started Store IsInteg task for mailbox '6e6a0983-02ab-4d1d-84ea-d0071e7e6536'. IsInteg RequestGuid = 'd783655f-7e84-4afb-b6a3-0669e750041a'.
    18/03/2017 12:53:33 AM [EXCH16-SYD-01] Store IsInteg task is pending completion for mailbox '6e6a0983-02ab-4d1d-84ea-d0071e7e6536'. IsIntegRequestGuid = 'd783655f-7e84-4afb-b6a3-0669e750041a'. Percentages complete: 0 .
    18/03/2017 12:54:03 AM [EXCH16-SYD-01] Store IsInteg task is pending completion for mailbox '6e6a0983-02ab-4d1d-84ea-d0071e7e6536'. IsIntegRequestGuid = 'd783655f-7e84-4afb-b6a3-0669e750041a'. Percentages complete: 0 .

Here's an excerpt of a move request that worked, going from 2013 to 2016:

   2/04/2017 8:22:30 PM [EXCH16-SYD-01] Connected to source mailbox '9a45c915-a87e-4211-8539-35034be5c0eb (Primary)', database

   ...

   2/04/2017 8:22:30 PM [EXCH16-SYD-01] Request processing started.
   2/04/2017 8:22:30 PM [EXCH16-SYD-01] Source mailbox information:
   Regular Items: 28128, 4.865 GB (5,223,415,468 bytes)
   Regular Deleted Items: 134, 8.387 MB (8,794,863 bytes)
   FAI Items: 440, 2.627 MB (2,754,276 bytes)
   FAI Deleted Items: 0, 0 B (0 bytes)
   2/04/2017 8:22:30 PM [EXCH16-SYD-01] Source archive mailbox information:
   Regular Items: 52386, 11.77 GB (12,639,788,901 bytes)
   Regular Deleted Items: 0, 0 B (0 bytes)
   FAI Items: 1, 2.658 KB (2,722 bytes)
   FAI Deleted Items: 0, 0 B (0 bytes)
   2/04/2017 8:22:30 PM [EXCH16-SYD-01] Cleared sync state for request
   9a45c915-a87e-4211-8539-35034be5c0eb due to 'CleanupOrphanedMailbox'.
   2/04/2017 8:22:30 PM [EXCH16-SYD-01] Cleared sync state for request
   f33ab71d-170b-4d55-bbfb-a9bb65db2fdc due to 'CleanupOrphanedMailbox'.
   2/04/2017 8:22:31 PM [EXCH16-SYD-01] Stage: CreatingFolderHierarchy.
   Percent complete: 10.
   2/04/2017 8:22:32 PM [EXCH16-SYD-01] Initializing folder hierarchy from
   mailbox '9a45c915-a87e-4211-8539-35034be5c0eb (Primary)': 1146 folders
   total.
   2/04/2017 8:22:32 PM [EXCH16-SYD-01] Folder creation progress: 0 folders
   created in mailbox '9a45c915-a87e-4211-8539-35034be5c0eb (Primary)'.
   2/04/2017 8:23:02 PM [EXCH16-SYD-01] Folder hierarchy initialized for
   mailbox '9a45c915-a87e-4211-8539-35034be5c0eb (Primary)': 1145 folders
   created.
   2/04/2017 8:23:02 PM [EXCH16-SYD-01] Initializing folder hierarchy from
   mailbox 'f33ab71d-170b-4d55-bbfb-a9bb65db2fdc (Archive)': 846 folders total.
   2/04/2017 8:23:22 PM [EXCH16-SYD-01] Folder hierarchy initialized for
   mailbox 'f33ab71d-170b-4d55-bbfb-a9bb65db2fdc (Archive)': 845 folders
   created.
   2/04/2017 8:23:22 PM [EXCH16-SYD-01] Stage: CreatingFolderHierarchy.
   Percent complete: 10.
   2/04/2017 8:23:22 PM [EXCH16-SYD-01] Stage: CreatingInitialSyncCheckpoint.
   Percent complete: 15.
   2/04/2017 8:23:22 PM [EXCH16-SYD-01] Initial sync checkpoint progress:
   0/1146 folders processed. Currently processing mailbox
   '9a45c915-a87e-4211-8539-35034be5c0eb (Primary)'.
   2/04/2017 8:24:06 PM [EXCH16-SYD-01] Initial sync checkpoint completed:
   1970 folders processed.
   2/04/2017 8:24:06 PM [EXCH16-SYD-01] Stage: LoadingMessages. Percent
   complete: 20.
   2/04/2017 8:24:06 PM [EXCH16-SYD-01] Stage: LoadingMessages. Percent
   complete: 20.
   2/04/2017 8:29:06 PM [EXCH16-SYD-01] Stage: CopyingMessages. Percent
   complete: 27.
   2/04/2017 8:29:06 PM [EXCH16-SYD-01] Copy progress: 4546/81089 messages,
   653.3 MB (684,996,208 bytes)/16.65 GB (17,874,496,811 bytes), 11/846
   folders completed.
   2/04/2017 8:34:10 PM [EXCH16-SYD-01] Stage: CopyingMessages. Percent
   complete: 28.
   2/04/2017 8:34:10 PM [EXCH16-SYD-01] Copy progress: 5536/81089 messages,
   818.5 MB (858,269,612 bytes)/16.65 GB (17,874,496,811 bytes), 11/846

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.