I'm having a problem between an older W2k3 application client and a manager server running on W2k12R2. I've tested the same W2k3 client to another manager also on W2k12r2 and have no issues so It appears the W2k3 server isn't the problem.
After taking packet captures on both the working and non working W2k12 servers along with the W2k3 server as I try and force the client to connect I'm seeing the usual Syn/Ack/SynAck and then Client Hello but then the server responds with an immediate RST and kills the connection. This is being picked up in the W2k12 system event log as an Schannel issue.
Server2012R2 36874 Error Schannel System 10/26/2017 4:39:08 PM An TLS 1.0 connection request was received from a remote client application, but none of the cipher suites supported by the client application are supported by the server. The SSL connection request has failed.
Server2012R2 36888 Error Schannel System 10/26/2017 4:39:08 PM A fatal alert was generated and sent to the remote endpoint. This may result in termination of the connection. The TLS protocol defined fatal error code is 40. The Windows SChannel error state is 1205.
I've verified ISSCrypto that the TLS1.0 is enabled on the W2k12R2 server as well as verified all cipher keys/suites match up correctly.
Using OpenSSL I've attempted a few tests and the only difference I see is on the Non working W2k12R2 servers the Public Key size is 4k where as on the working one (using a different cert) it's a Public Key size of 2k.
My question is can Windows 2003 SP2 handle a 4k key size? Also were there patches that addressed the key size around the time of POODLE?
Any assistance will be greatly appreciated.
