You're on the right track. You have to use certutil.exe to create the certificate, then you'll see it in the drop-down in IIS Management in the edit bindings window.
Instead of importing a private key, use the capability of certutil.exe to generate and save the private key in the Windows certificate store, and then match it to the issued certificate.
To do this, one pain point is creating the request.inf file. Here is a sample that worked for me on Windows Server 2012 R2 Core for WSUS. Fill in the FQDN with the name of your server.
Subject = "CN=FQDN"
Exportable = FALSE
KeyLength = 2048
KeySpec = 1
KeyUsage = 0xA0
MachineKeySet = True
ProviderName = "Microsoft RSA SChannel Cryptographic Provider"
RequestType = PKCS10
OID=126.96.36.199.188.8.131.52.1 ; Server Authentication
OID=184.108.40.206.220.127.116.11.2 ; Client Authentication
CertificateTemplate = Machine
At the command prompt of the core server, in the folder where request.inf is, run:
certreq -new request.inf machine.req
Then, submit the machine.req file to your CA. If you have Windows Enterprise CA, you can do it this with this command:
certutil -submit -adminforcemachine machine.req machine.cer
When you get the certificate (either from your Enterprise CA or third-party), import it into the certificate store.
certreq -accept machine.cer
Then, go into IIS Manager on your admin workstation, connect to the WSUS server, and bind the WSUS Administration site to your certificate.
Then, you can enable "Require SSL" for:
Don't do it for any other virtual folder.
According to the documentation, you're supposed to run this on the WSUS server:
"C:\Program Files\Update Services\Tools\wsusutil" configuressl <certificate>
But, I'm not sure this is necessary.
Hope this helps!