TLS 1.2 client hello triggers TCP Reset from 2012 R2

Question

Struggling with a PKI implementation in my lab (ADCS 2012 R2) and cannot for the life of me figure out where I have gone wrong.

Got all the way to the point of being able to generate SSL/TLS certs for an IIS server and make the binding. Also used the NARTAC tool to shut down SSLv2/3 + enable TLS 1.0-1.2. Using testssl.sh I can verify that SSL is disabled and can get a handshake for TLS 1.0 and 1.1, but every time the client hello for TLS 1.2 is received the server sends a TCP Reset. The system logs show the pair of SCHANNEL 36874 and 36888 error IDs that correspond with the resets. Error 36874 suggests that the client hello is presenting cipher suites that aren't supported by the server.

This is where I hit the brick wall, and my searching the interwebs is not yielding a winner. I can all but guarantee I've overlooked something glaringly obvious. However, the blinders of frustration are preventing me from seeing it.

Thoughts?

Client hello details:

Frame 136: 377 bytes on wire (3016 bits), 377 bytes captured (3016 bits) on interface 0 Ethernet II, Src: CiscoInc_07:be:7f (fc:5b:39:07:be:7f), Dst: Vmware_01:02:14 (00:50:56:01:02:14) Internet Protocol Version 4, Src: 192.168.6.75, Dst: 10.22.163.219 Transmission Control Protocol, Src Port: 35836 (35836), Dst Port: 443 (443), Seq: 1, Ack: 1, Len: 311 Secure Sockets Layer SSL Record Layer: Handshake Protocol: Client Hello Content Type: Handshake (22) Version: TLS 1.0 (0x0301) Length: 306 Handshake Protocol: Client Hello Handshake Type: Client Hello (1) Length: 302 Version: TLS 1.2 (0x0303) Random Session ID Length: 0 Cipher Suites Length: 176 Cipher Suites (88 suites) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030) Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028) Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014) Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a) Cipher Suite: TLS_DH_DSS_WITH_AES_256_GCM_SHA384 (0x00a5) Cipher Suite: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384 (0x00a3) Cipher Suite: TLS_DH_RSA_WITH_AES_256_GCM_SHA384 (0x00a1) Cipher Suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f) Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (0x006b) Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 (0x006a) Cipher Suite: TLS_DH_RSA_WITH_AES_256_CBC_SHA256 (0x0069) Cipher Suite: TLS_DH_DSS_WITH_AES_256_CBC_SHA256 (0x0068) Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039) Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038) Cipher Suite: TLS_DH_RSA_WITH_AES_256_CBC_SHA (0x0037) Cipher Suite: TLS_DH_DSS_WITH_AES_256_CBC_SHA (0x0036) Cipher Suite: TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA (0x0088) Cipher Suite: TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA (0x0087) Cipher Suite: TLS_DH_RSA_WITH_CAMELLIA_256_CBC_SHA (0x0086) Cipher Suite: TLS_DH_DSS_WITH_CAMELLIA_256_CBC_SHA (0x0085) Cipher Suite: TLS_GOSTR341001_WITH_28147_CNT_IMIT (0x0081) Cipher Suite: TLS_GOSTR341094_WITH_28147_CNT_IMIT (0x0080) Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 (0xc032) Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02e) Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 (0xc02a) Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 (0xc026) Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA (0xc00f) Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA (0xc005) Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d) Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d) Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035) Cipher Suite: TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (0x0084) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f) Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027) Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009) Cipher Suite: TLS_DH_DSS_WITH_AES_128_GCM_SHA256 (0x00a4) Cipher Suite: TLS_DHE_DSS_WITH_AES_128_GCM_SHA256 (0x00a2) Cipher Suite: TLS_DH_RSA_WITH_AES_128_GCM_SHA256 (0x00a0) Cipher Suite: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e) Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (0x0067) Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 (0x0040) Cipher Suite: TLS_DH_RSA_WITH_AES_128_CBC_SHA256 (0x003f) Cipher Suite: TLS_DH_DSS_WITH_AES_128_CBC_SHA256 (0x003e) Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033) Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032) Cipher Suite: TLS_DH_RSA_WITH_AES_128_CBC_SHA (0x0031) Cipher Suite: TLS_DH_DSS_WITH_AES_128_CBC_SHA (0x0030) Cipher Suite: TLS_DHE_RSA_WITH_SEED_CBC_SHA (0x009a) Cipher Suite: TLS_DHE_DSS_WITH_SEED_CBC_SHA (0x0099) Cipher Suite: TLS_DH_RSA_WITH_SEED_CBC_SHA (0x0098) Cipher Suite: TLS_DH_DSS_WITH_SEED_CBC_SHA (0x0097) Cipher Suite: TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA (0x0045) Cipher Suite: TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA (0x0044) Cipher Suite: TLS_DH_RSA_WITH_CAMELLIA_128_CBC_SHA (0x0043) Cipher Suite: TLS_DH_DSS_WITH_CAMELLIA_128_CBC_SHA (0x0042) Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 (0xc031) Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02d) Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 (0xc029) Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 (0xc025) Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA (0xc00e) Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA (0xc004) Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c) Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c) Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f) Cipher Suite: TLS_RSA_WITH_SEED_CBC_SHA (0x0096) Cipher Suite: TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (0x0041) Cipher Suite: TLS_RSA_WITH_IDEA_CBC_SHA (0x0007) Cipher Suite: TLS_ECDHE_RSA_WITH_RC4_128_SHA (0xc011) Cipher Suite: TLS_ECDHE_ECDSA_WITH_RC4_128_SHA (0xc007) Cipher Suite: TLS_ECDH_RSA_WITH_RC4_128_SHA (0xc00c) Cipher Suite: TLS_ECDH_ECDSA_WITH_RC4_128_SHA (0xc002) Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005) Cipher Suite: TLS_RSA_WITH_RC4_128_MD5 (0x0004) Cipher Suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (0xc012) Cipher Suite: TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc008) Cipher Suite: TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (0x0016) Cipher Suite: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013) Cipher Suite: TLS_DH_RSA_WITH_3DES_EDE_CBC_SHA (0x0010) Cipher Suite: TLS_DH_DSS_WITH_3DES_EDE_CBC_SHA (0x000d) Cipher Suite: TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA (0xc00d) Cipher Suite: TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc003) Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a) Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff) Compression Methods Length: 1 Compression Methods (1 method) Extensions Length: 85 Extension: ec_point_formats Extension: elliptic_curves Extension: SessionTicket TLS Extension: signature_algorithms Type: signature_algorithms (0x000d) Length: 32 Signature Hash Algorithms Length: 30 Signature Hash Algorithms (15 algorithms) Extension: Heartbeat

And the TCP reset:

137 2.086290 10.22.163.219 192.168.6.75 TCP 54 443 → 35836 [RST, ACK] Seq=1 Ack=312 Win=0 Len=0 Frame 137: 54 bytes on wire (432 bits), 54 bytes captured (432 bits) on interface 0 Ethernet II, Src: Vmware_01:02:14 (00:50:56:01:02:14), Dst: IETF-VRRP-VRID_19 (00:00:5e:00:01:19) Internet Protocol Version 4, Src: 10.22.163.219, Dst: 192.168.6.75 Transmission Control Protocol, Src Port: 443 (443), Dst Port: 35836 (35836), Seq: 1, Ack: 312, Len: 0 Source Port: 443 Destination Port: 35836 [Stream index: 3] [TCP Segment Len: 0] Sequence number: 1 (relative sequence number) Acknowledgment number: 312 (relative ack number) Header Length: 20 bytes Flags: 0x014 (RST, ACK) Window size value: 0 [Calculated window size: 0] [Window size scaling factor: 256] Checksum: 0x74ff [validation disabled] Urgent pointer: 0 [SEQ/ACK analysis]

SCHANNEL 36874

Log Name: System Source: Schannel Date: 5/4/2016 9:48:36 AM Event ID: 36874 Task Category: None Level:
Error Keywords: User: SYSTEM Computer:
OCSP.corp.contoso.com Description: An TLS 1.2 connection request was received from a remote client application, but none of the cipher suites supported by the client application are supported by the server. The SSL connection request has failed.

SCHANNEL 36888:

Log Name: System Source: Schannel Date: 5/4/2016 9:48:36 AM Event ID: 36888 Task Category: None Level:
Error Keywords: User: SYSTEM Computer:
OCSP.corp.contoso.com Description: A fatal alert was generated and sent to the remote endpoint. This may result in termination of the connection. The TLS protocol defined fatal error code is 40. The Windows SChannel error state is 1205.

EDIT 1

Found that I can successfully get a non-domain joined 2012 R2 IIS server to respond to a TLS 1.2 hello packet. So my issue appears to be related to being a domain joined system. Still haven't solved that part of the equation.

Answer 1

Certain ciphers seem to have interoperability issues. We had the same problem. The problematic ciphers we found are:

• TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
• TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e)
• TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
• TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
• TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
• TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)

See also these links.

• https://docs.microsoft.com/en-us/archive/blogs/keithab/transport-layer-security-tls-handshake-failing-schannel-error-36888
• https://support.microsoft.com/en-us/help/2992611/ms14-066-vulnerability-in-schannel-could-allow-remote-code-execution-n

Answer 2

If the issue is a domain joined computer not being able to connect (and I assume the connection client is Windows itself, IE, or another component that makes use of the native SChannel in Windows), then I would suspect that you don't have TLS 1.2 enabled on the client. And being a domain joined computer, this would mean you have a GPO Administrative Templates that's configuring the protocol and cipher suites and pushing them to the client.

Couple things I'd recommend for tracing.

I assume you're mention of the "Nartac tool' means IISCrypto. Run this up on both the client and server to see what protocols and ciphers are enable - ensure there's overlap.

Also, any changes to these items requires a reboot. Yes, this may be stating the obvious, but it needed to be said.

Next, get Fiddler. This tool is my best friend. You can use it to trace http and https traffic from a client (it inserts itself as a middle man between client and server, acting as a proxy for the client, decrypts https traffic, etc). The important part is that it also traces the https connect (and reveals cipher suits, etc). Run it on your client and see what it says.

My only other thought is that the AD Cert Authority you've setup, the root cert certificate hasn't made it into the Trusted Root Certification Authorities certificate store on the client computers. Check this too.

Answer 3

Another possible cause of RST could be signature_algorithm extension being missing (in which case defaults to SHA1) or not having a pair that matches the server cert signature. So if you upgrade to a sha512 RSA signed cert and the client only offers SHA1 and SHA256 the server will RST your client hello.

Not the case here but if anyone find this question, this may help.