I know, this is quite old question, but I facing similar issue (need to remove private key) and here is the solution
Hi, This applies for all MS Windows versions.
If you have a certificate installed on HD (i.e. using the MS Enhaced
CSP), then, following Microsoft, you can remove it using IExplorer, on
the Tools menu, you click Internet Options, then you click the*
Content* tab, and then click Remove. This is a well known action
Doing this, you effectively remove the certificate, but THE PRIVATE
KEY REMAINS IN THE HD. You can find a lot of scenarios where this can
be a problem. Suppose you go to a friend's home, you install a pkcs12
file containing your certificate and private key with "Medium"
security level (the default), then you use it, and when you finishes
your work, you remove the certificate (but NO the private key). Then
your friend takes your certificate (is a public document) and installs
it, having your private key working for him.
The program cleancapi deletes the private keys that are not used by
any certificate. Source code: