CA Recovery in Active Directory

Question

I think I understand, but I want to be sure.

We are running two Windows 2016 domain controllers (VMs), one of which is performing the role of a certificate server. We perform system state backups daily and offload them to a remote location. If, for whatever reason, the Domain Controller with the CA needs to be completely rebuilt, I assume I can do a non-authoritative restore using the most recent system state backup to recover the CA and the existing domain controller will update the rebuilt server with all other AD information. Is this a correct assumption or should I implement a CA specific recovery plan?

Additional Information: I am working on a baseline for a system deployed in 50+ disconnected environments. Adding another server to each site is not an option for me at this point. I plan on implementing both system state and a CA specific backup plan to aid in recovery, here are a few scripts I scrapped together to aid in manual checks and automated daily processes.

#CA Initial / Update Backup Script
$filedate = (Get-Date -format d).ToString().Replace(“/”,”-“)

#Backup / verify backup
IF ((Test-Path D:\CAbackup) -eq $False)
    {
    mkdir D:\CABackup
    mkdir D:\CABackup\InitialBackup
    Backup-CARoleService -KeyOnly D:\CABackup\InitialBackup -Password (read-host -Prompt "Assign a password for the CA Private Key" -AsSecureString)
    Backup-CARoleService -DatabaseOnly D:\CABackup\InitialBackup
    reg export HKLM\System\CurrentControlSet\Services\CertSvc\Configuration D:\CABackup\InitialBackUp\<filename>.reg
    }
 ELSEIF ((Test-Path D:\CABackup\InitialBackup) -eq $False)
    {
    mkdir D:\CABackup\InitialBackup
    Backup-CARoleService -KeyOnly D:\CABackup\InitialBackup -Password (read-host -Prompt "Assign a password for the CA Private Key" -AsSecureString)
    Backup-CARoleService -DatabaseOnly D:\CABackup\InitialBackup
    reg export HKLM\System\CurrentControlSet\Services\CertSvc\Configuration D:\CABackup\InitialBackUp\<filename>.reg
    }
ELSE {
    IF ((Test-Path D:\CABackup\InitialBackup\database\<filename>.edb) -eq $false)
        {
        Backup-CARoleService -DatabaseOnly D:\CABackup\InitialBackup
        }
    ELSE {}
    IF ((Test-Path D:\CABackup\InitialBackup\<filename>.reg) -eq $false)
        {
        reg export HKLM\System\CurrentControlSet\Services\CertSvc\Configuration D:\CABackup\InitialBackUp\<filename>.reg
        }
    ELSE {}
    IF ((Test-Path D:\CABackup\InitialBackup\<filename>.p12) -eq $false)
        {
        Backup-CARoleService -KeyOnly D:\CABackup\InitialBackup -Password (read-host -Prompt "Assign a password for the CA Private Key" -AsSecureString)
        }
    ELSE
        {
        #Compare the backed up certificate thumbprint against the CA certificate thumbprint, if they do not match, archive the old cert and back up the current one
        $catpret = certutil -adca | Select-String "Cert Hash"
        $catp = $certret.Line.Substring(17)
        $archtp = (Get-PfxData -FilePath d:\cabackup\Prikey\<filename>.p12).EndEntityCertificates.Thumbprint
        IF ($catp -ne $archtp)
            {
            mv d:\cabackup\prikey\<filename>.p12 d:\cabackup\prikey\<filename>.p12.$filedate
            Backup-CARoleService -DatabaseOnly D:\CABackup\InitialBackup
            }
        ELSE {}
        }
    }

#List of certificates that will expire in next 120 days
$list=@()
$na =(get-date).addDays(120)
$listofexp = certutil -view -restrict "NotAfter<=$na" -out "RequestID,RequesterName,Request Common Name,NotAfter"
$total = ($listofexp.count -10)
$f=10
$e=13
While ($e -lt $total)
    {
    $list += ($listofexp[$f] + $listofexp[$f+1] + $listofexp[$f+2] + $listofexp[$e])
    $f = $f+6
    $e = $e+6
    } 


#Daily backup
$filedate = (Get-Date -format d).ToString().Replace(“/”,”-“)
mkdir D:\CABackup\$filedate
Backup-CARoleService -DatabaseOnly D:\CABackup\$filedate
reg export HKLM\System\CurrentControlSet\Services\CertSvc\Configuration D:\CABackup\$fildate\<filename>.reg

#Clear 2 week and older cert requests
$list=@()
$setpurge = (get-date).AddDays(-14)
$purgedate = Get-date $setpurge -Format "MM/dd/yy"
$listofpend = certutil -view -restrict "Request Submission Date<=$purgedate,Request Disposition=9" -out "Request ID, Request Submission  Date, Request Common Name"
$total = ($listofpend.count -9)
$f=9
$e=11
While ($e -lt $total)
{
$list += ($listofpend[$f] -replace '.*\(' -replace '\),*')
$f = $f+5
$e = $e+5
}
foreach ($item in $list)
{
certutil -deleterow $item
}

<#Original purge process replaced by above
$setpurge = (get-date).AddDays(-14)
    $purgedate = Get-date $setpurge -Format "MM/dd/yy"
    certutil -deleterow $purgedate request
#>

I have tested most of this out in a lab and am letting the dailies run for a bit before I delete a server or two and attempt restoration. If anyone has any additional advice it would be greatly appreciated.

Answer 1

This is one of the reasons why it's highly recommended that you do not have a CA running on a domain controller. I have one on a DC in a legacy domain right now and it is a maintainability nightmare and it will be removed when another service dependency is sorted out. DCs and CA servers are highly critical, of course, and if you have one function fail while the other is fine, it makes it much more complex to deal with.

I highly recommend implementing a separate maintenance plan to backup your CA database via a scheduled task. This maintenance plan should also include processes (certutil or PKPSI Powershell) to regularly delete old rejected or pending certificate requests plus expired certificates. Not only will it help if you want to migrate, or recover the CA elsewhere, you should do a regular backup like any database to clean up the transaction logs and ensure everything is committed.

To get one going, do a full backup of the CA - including the CA key - to a directory on a local volume that doesn't have CA database on it. Call the directory something like CABackup, then create a child directory that says something like "Initial Backup". Target your first backup there. The CA private key should be archived somewhere secure with a record of the password used to save it. Also export the registry key HKLM:\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration\[CAName] to your backup folder.

For regular maintenance, we have a script that creates a new backup folder underneath the "CABackup" parent folder each time it runs - the directory must be empty for each backup. It backs up the CA database only. Naturally, the CA backup directory and its contents is archived to tape with the rest of the regular filesystem backup. The same script also performs the cleanup of stale REQs and expired certs if the local backup is successful.

You should have a process to delete old backups - we keep the last 5 available - but don't delete your Initial Backup folder. Of course, if you change the CA key for some reason, do a new Initial Backup of both the DB and CA key.

In the domain where a CA and DC coexist, in the event the entire domain needed recovery, I would not use the DC hosting the CA as a recovery target. Any alternate DC would be preferable. If the actual domain is healthy but something was wrong with ADDS on the shared server, I would simply remove ADDS and build another DC.

If the CA required recovery, but not the DC, I'd take the opportunity to migrate the CA. The exported reg key helps expedite the process, but ensure you configure the recovered CA instance with the new server FQDN in the Registry. This procedure to migrate a CA describes the process. Also, naturally, test the CA recovery process in an isolated environment.

It might be perfectly fine recovering both DC and CA with a system state recovery, but to be honest, one or other of those is stressful enough by itself. I've only done it myself once in very long-ago days in an environment with a single "everything" DC/CA/file server.

I do recommend moving your CA as soon as possible, though, before you incur much more "technical debt". It can coexist with some other less-critical roles, like a KMS server or WSUS, if you have constraints on the number of systems you can have. If you aren't constrained, you should do a two-tier offline root plus intermediate CA setup. The migration article linked above is worth a read even just for your info - a basic migration isn't a very difficult process.

Also, what you should definitely do, if you keep your current config (for a while) is as soon as possible try a system state recovery in an isolated environment, including adding at least one new DC to the recovered domain and ensuring the CA can issue certs to the kind of clients it does currently. That should help clarify any potential pain points.