Last Saturday I had one of the Citrix servers in my cluster start complaining about losing the trust relationship with the domain. While this is not really common, it's not unheard of. Usually, the process is reset AD computer account, remove from the domain, re-join to the domain. Easy peasy.
With this server, however, the tables have turned. I followed my normal process of resetting the computer account and removing from the domain using a local admin account, but when I rebooted the server was still looking for the domain. Tried again thinking I had messed up somehow and received the same result.
I then tried restoring the server from a backup the night before (when I originally looked I had misread the backup logs and thought that particular server hadn't been backed up for a while; not uncommon with terminal servers) and experienced the same results. I even tried sysprepping the machine and after rebooting it came back up to the domain login prompt.
A friend suggested I try creating a new local admin account and removing from the domain that way. I created the account, rebooted, then the account was gone. I created it again and just logged out, then back in as the new admin and that got me in. I removed from the domain and rebooted with predictable results. No trace of the user account I had created and successfully logged in with. No user directory, nothing in the registry for profile list, nothing. After this happened I ran chkdsk and sfc /scannow just for giggles and both came up with no errors.
This is a Windows Server 2012 R2 running on VMware 6.5. I wouldn't put this much effort in normally when I could clone/sysprep/rejoin one of the other servers, but this server has a particular set of users with specific requirements. I repurposed another server for them in the meantime, but ideally, I'd like to get this one back into service.
When the event initially happened I did check the system time and the logs and found nothing out of the ordinary. The server was dead on with both of the domain controllers.