16

I was recently in an environment where there were 120 domain controllers in over 100 sites around the world. This domain began in the Windows 2000 era and was upgraded over time, so strict replication consistency was never set as the default for new DCs and has never been enabled on any DCs. There are lingering objects in the directory and you regularly see a decent number of conflicted objects because of this.

Using repadmin /removelingeringobjects requires knowing two things:

  1. Which DC(s) have lingering objects in the database

  2. A DC that has no lingering objects on it to use as a reference DC.

Obviously, in the future, this environment should be set so that all new DCs have strict replication consistency enforced and repadmin /options * +strict should be run to make all current DCs use strict replication consistency, but that will break replication now without cleaning the objects.

So, my question is this: In such a massive environment where I would have no idea which DCs have lingering objects and which ones do not, how can I identify a good reference DC for repadmin /removelingeringobjects to use, and how can I ensure that all 120+ DCs are clean of lingering objects before enforcing strict replication consistency and breaking replication? Or, is it just easier to turn on strict mode and watch repadmin /replsum to see what breaks and deal with it?

11

This will take some time to fix.

To stop all replication, run:

repadmin /options +DISABLE_OUTBOUND_REPL

On all DCs. Remember that the above setting does not prevent manual replication actions such as an admin (you) running repadmin /syncall /APed, etc. But that's a good thing because it allows you to get all your DCs back into sync totally before re-enabling regular replication.

Repadmin determines that it's a lingering object if the object exists on ServerA but not on ServerB, where ServerB is the reference DC. The difference between replicating newly created objects and replicating updates to already existing objects is the key. Replicating newly created objects = good. Replicating updates to already existing objects = good. Replicating updates to objects that don't exist on the destination DC = bad.

You only need to lather, rinse, repeat until all DCs match up with your one good reference DC. Then turn on strict consistency everywhere, then re-enable replication. Yes, you do run the risk of deleting legitimate objects that were created on other remote DCs that have not replicated to your reference DC.

From the great "How the Active Directory Replication Model Works" article:

Replication Consistency Setting

If the attributes on a lingering object never change, the object is never considered for replication. However, if an attribute changes, the attribute is considered for outbound replication. Because the destination domain controller does not hold the object for the attribute that is being replicated, an update cannot be performed. How this condition is resolved depends on the replication consistency setting on the domain controller.

A registry setting on domain controllers that are running Windows Server 2003 or Windows 2000 Server with SP3 provides a consistency value that determines whether a domain controller replicates and reanimates an updated object that has been deleted from all other replicas, or whether replication of such objects is blocked. The default settings are different on domain controllers that are running Windows 2000 Server with SP3 and Windows Server 2003.

Strict Replication Consistency

To avoid problems with reanimating objects that have been deleted, a domain controller that is running Windows Server 2003 in a newly created (not upgraded) Windows Server 2003 forest blocks inbound replication by default when it receives an update to an object that it does not have.

Note • Active Directory replication uses update tracking to differentiate between replicating a newly created object and updating an attribute for an existing object. Replication of a lingering object is an attempt to update an attribute or attributes of an object that the destination domain controller cannot update because the object does not exist.

Replication is halted in the directory partition for the object until the lingering object is removed from the source domain controller or the strict replication consistency setting is disabled.

When ServerB says to ServerA: "Hey, some updates have been made to existing objectA." Then ServerA says: "Wait what? I don't even have objectA at all. Send me the entire object!" If no strict consistency. If strict consistency, ServerA says: "Wait what? How do you expect me to update an object that doesn't exist? Go get bent!"

To find if you have lingering objects on a domain controller:

repadmin /removelingeringobjects ServerName ServerGUID DirectoryPartition /advisory_mode 

ServerGUID is the known good reference DC. I know you already know this... and how to script the above line to run it on all DCs... (foreach ($DC In $(Get-ADDomain).ReplicaDirectoryServers) { })...

You need a good source DC to compare against, bottom line. If you don't have a known good source DC or don't know, you just have to pick one. It should be a writable GC of course. It's relative - if all domain controllers agree on the existence of an object, and that object's attributes... then it's not a lingering object.

foreach($GC In $(Get-ADForest).GlobalCatalogs) { repadmin /removelingeringobjects $_.name 85d158d2-a006-4fff-b1e5-f9b6eaabab2b '$directoryPartition'

That's resyncing that directory partition of every GC in the forest with the known good source which you need to specify as the GUID.

Then after you've got all your domain controllers once again all in agreement, and replication is happy... then you go and start flipping on strict consistency on all of them.

Edit: This is Microsoft's party line on the issue, and what they'll likely talk you through were you to call them.

Finally, this may be more trouble to fix than it's worth unless it's causing you problems. I hate to say it, but AD can still function normally with lingering objects in it.

4

The general principle in a case where you can't identify a clean DC is as follows:

for each $sourceDC in $allDCs
    for each $targetDC in $allDCs
        if ($targetDC <> $sourceDC) then
            run repadmin with $sourceDC and $targetDC
        end if
    next
next

This process is described here: http://blogs.technet.com/b/glennl/archive/2007/07/26/clean-that-active-directory-forest-of-lingering-objects.aspx

However, take a look at ReplDiag. It automates the process by running repadmin for you against all combinations of source and target DC's. It then follows up with /advisory_only to check for any further lingering objects.

  • That's still 14,400 interations that this will run through, during which time, additional lingering objects can be replicated throughout the directory. Is it recommended to enable +strict first so that replication breaks to avoid this? 14,400 iterations will definitely take long enough on a directory of this size to allow for more bad replication to take place. – MDMarra Aug 8 '13 at 15:07
  • Can you parallelize it with something? – Tom O'Connor Aug 8 '13 at 15:22
  • @TomO'Connor Interesting idea. That should be possible with PowerShell remoting. Then you could get each DC to run repadmin for itself, in PARALLEL!!! – longneck Aug 8 '13 at 15:29
  • 7
    @MDMarra If it was quick and easy to clean a 120-DC domain that spans the globe and has been in place since Win2k and upgraded throughout various versions since then... then the company would just get their janitor to do it... ;) – Ryan Ries Aug 8 '13 at 15:32
  • 1
    I realize you have a bazillion DC's, but how many of them are actually used by administrators to make changes? You could start by doing this cleanup with just those DC's and the DC's in those sites, then set strict on those DC's. The continue with the others. That should minimize any possible problems – longneck Aug 8 '13 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.