Windows Server 2008 R2 - RDSH - Suddenly Windows Activation Lost

Question

On 2014-06-17, we had eight of our Windows Server 2008 R2 RDSH Servers (all members of the same RDSH farm {brokered}) suddenly believe Windows Activation was lost while users were logged in. If user was previously logged in they were somewhat functioning, but nobody could establish a new logon. If an Administrator tried to Activate Windows the error code 0x80070005 would occur. Each of these servers is running ESET File Security for Windows Servers 4.5.12011.0 for anti-virus protection. Event Viewer would not load at all and would crash MMC (eventually found Event ID 6001 & 6004 from Winlogon occurred right when they all went crazy).

1. Searched TechNet and found: http://support.microsoft.com/kb/2704233
2. Reviewed the registry and found the following keys were completely missing:
   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Profilelist\S-1-5-18
   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Profilelist\S-1-5-19
   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Profilelist\S-1-5-20
3. Also found keys in "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Profilelist\" were missing:
   Default
   ProfilesDirectory
   ProgramData
   Public
4. After further investigation found Symbolic link for "Default User" in C:\Users\ was also missing(should link to C:\Users\Default and have special NTFS Folder Only privileges).
5. Found C:\Users\Public\Desktop was named "Public Desktop" though a Symbolic link from "C:\ProgramData\Desktop" was pointing to "C:\Users\Public\Desktop"

NOTE: I will publish what it took to fix, as I already resolved but wanted to post the info here on ServerFault.com in case somebody else runs into this.

Answer 1

Here is how I was able to resolve the issue. There are some EXTREMELY DANGEROUS and DATA DESTRUCTIVE steps here. If you DO NOT have Roaming Profiles & Folder Redirection of the User Accounts than you seriously should call Microsoft Support as this is NOT the way to fix your issue as it will remove files!!!!! ANY USE OF THESE INSTRUCTIONS IS AT YOUR OWN RISK!!!!!

1. Set the server to not allow new logons in RD Session Host Configuration (User logon mode)
2. Logged users off the server (gracefully where possible)
3. Recreated missing profilelist registry keys (See MS Article):
   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Profilelist\S-1-5-18
   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Profilelist\S-1-5-19
   HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Profilelist\S-1-5-20
4. Recreated HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\ keys (REG_EXPAND_SZ)
   Default > %SystemDrive%\Users\Default
   ProfilesDirectory > %SystemDrive%\Users
   ProgramData > %SystemDrive%\ProgramData
   Public > %SystemDrive%\Users\Public
5. Used Sysinternals Junction to review "C:\Users\Default User" and recreate "C:\Users\Default User" to symbolically link to "C:\Users\Default"
   Security - Advanced - Change Permissions...
   Uncheck "Include inheritable permissions..."
   Add "Everyone" > Apply to "This folder only" > Deny "List folder / read data"
   Add "Everyone" > Apply to "This folder only" > Allow "Traverse folder / execute file" + "List folder / read data" + "Read Attributes" + "Read extended attributes" + "Read permissions"
   Add "SYSTEM" > Apply to "This folder only" > Allow "Full Control"
   Add "Administrators (%COMPUTERNAME%\Administrators)" > Apply to "This folder only" > Allow "Full Control"
6. Set NTFS Privileges to "C:\Users\Default User" to match identically another RDSH server (from a different farm so I would know what the defaults are)
7. Renamed "C:\Users\Public\Public Desktop" to "C:\Users\Public\Desktop"
8. Restarted the server
9. After a restart Windows recognized that Windows was activated and services were running properly (Windows Defender had been bombed out before).
10. System Properties > Advanced > User Profiles > Was not showing any of the previous RDSH users.
11. We use Roaming Profiles and Folder Redirection GPOs so the Users not showing up in the Profiles was not a total loss. Just had to clean up ProfileList registry keys of the previous RDSH users in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Profilelist\ (Do not delete the local machine administrator %SID% & S-1-5-18, S-1-5-19, S-1-5-20 keys!!!!!)
12. Had to clean up C:\Users\ of the user folders where the ProfileList keys were removed.
    Don't do this if you do not have roaming profiles and folder redirection setup where the user files are not stored only on this server!!!
    Select All Files & Folders Except the following:
    All Users
    Administrator
    Administrator.{%YOURDOMAINNAME%}
    Default
    Default User
    Public
    desktop.ini
    
    Delete permanently the selected folders that are for Profiles that no longer exist in the ProfileList section of the registry (SHIFT+DELETE).
13. Restart again and verify everything is functioning.
14. Once you are happy with the server, allow new logons again in RD Session Host Configuration.

NOTE: If you are running virtual RDSH servers, you may want to restore your RDSH servers from backups if you have a properly working backup system if you want to avoid this craziness.