How well timed. I just got back from an emergency trip to one of the sites we support.
As far as user impact it wasn't a major impact but it had the potential to be.
As part of an ongoing project to migrate some sites off of our support we created a new trusted domain. After extensive testing we prepped for the first site to migrate to the new domain which we would still manage. So the night of the migrate comes along and we start by migrating one of two DCs to the new domain. That goes fine. We migrate the security groups and User accounts. That goes fine as well and group membership is updated properly. We migrate the File Server and run security translation to update the ACLs. Again all goes well. Migrate App servers and update IAS for VPN and no problems. We then migrate a test user PC and the user retains their profile settings and can access all network resources perfectly. We then migrate the other DC. We then go migrate the remaining computers and half fail. We find that the local XP firewall is on. I immediately push a GPO to the site to turn it off but will have to wait the computers refresh. This does not happen quickly enough and users start arriving. They can't log into the original domain because both DCs are on the new one.
Rather then try re-adding one DC back to the original domain we update firewall rules to allow access to other remote DCs for the original domain and take the 3 hour drive to the site.
Going on little sleep: The GPO to disable the local firewall has now pushed out. Without thinking i grab all the computer objects and push the migration. I forgot that this RESETs the computer objects. So now all the successfully migrated PCs are cut from the domain.
To make matters worse the local admin pass we roll out with our image does not work because of a long gone on-site tech that reset them.
I spent the weekend manually adding all the PCs to the new domain after using a boot disk to wipe the local admin pass.
Lessons Learned:
- Expect the unexpected and try to anticipate all potential problems.
- Centralize management of things like local admin passwords and firewall settings
- GPOs and Group Policy Preferences
- Take another minute to make sure you are doing things correctly before clicking.
Sorry that was long winded.