I'm taking care of a legacy win2k3 (sp2) which has IIS 6 installed. I have a very weird problem which is making me pull my hair out.
- There are 5 websites configured to run on the particular webserver.
- Of these, 3 have SSL certs that are functional (and their own dedicated IPs with SSL served on :443)
- One of these websites is having the following issue:
When the w3svc service is first started (via a reboot, or net stop/start), the initial connections to the SSL service of that particular website immediately fail with ERR_SSL_PROTOCOL_ERROR. I will provide further diagnostic information below.
Now for the list of completely baffling symptoms and attempted remedies:
- if I click the "stop" icon in the IIS MMC, and then start back up that website (not the w3svc service itself), the problem goes away.
- if I assign an SSL port other than 443 to the website, the problem never appears
- if I assign a different SSL cert to the website, the problem persists
- I created an entirely new website from scratch just to test this, the problem reappears
I can see no relevant log information in the event manager (Application, System, or Security) nor in the w3svc log files.
Filemon does not show anything weird that I can tell (access denied or file not found of any sort), regardless, I doubt the certificate is actually fetched from file.
I have run other tests using
wget and the
SSLDiagtools, all of which say more of the same without any real further diagnostic information.
A wire shark inspection reveals that the connection is simply dropped:
- 226 byte "Client Hello" packet is sent by client to server
- A [FIN,ACK] packet is immediately sent back by the server to the client
When the server is "remedied" using the stop/start method outlined above, the server responds normally (no such FIN/ACK packet is sent).
The above makes me wonder if it's a sudden page fault'y kind of thing that immediately terminates the thread before any logging or anything gets done.
I am at a loss, I have spent over 8 hours on this bug and still can not find a way to tackle it. I would love it if someone actually knew what was happening, but I'm actually just hoping for some ideas on how to further debug/inspect to determine the source of the problem.
NB: There are constraints which I can not avoid, like reinstalling the server or upgrading the server are not options at the moment.