Replacing SSL certificate for Nginx - inaccessible from existing devices

Question

I am really hoping someone with more experience can help me get our production site back online as i am pulling my hair out!

I have just renewed an expired ssl cert on our production website, I created a new certificate and key then ordered a new ssl on Comodo. Since I added the new comodo signed certificate and key I can't connect to our production website from any device that has previously used the website with the old certificate. However on an iPad or iPhone everything works as should.

If i try and connect all i see in the bottom left corner is Performing TLS handshake and then i get the page is insecure error, I have checked online tools like: https://www.ssllabs.com/ssltest/ and they say the ssl is fine and valid until February 2018.

Below is my config file for nginx however I am starting to think this is related to the browser caching the old certificate as i have rebooted the server several times to see if that would change anything.

server {
    listen       80 default_server;
    listen       [::]:80 default_server;    
    server_name  _;
    server_tokens off;
    return 301 https://$host$request_uri;

}

server {
    listen       443;
    server_name  *;
    server_tokens off;
    ssl    on;
    ssl_certificate /jet/etc/nginx/ssl/cert.crt;
    ssl_certificate_key /jet/etc/nginx/ssl/key.key; 

    error_page  403   https://$host/404;
    error_page  404   https://$host/404;

    root         /jet/app/www/public;


    index index.php index.html index.htm;
    autoindex on;

    # Load configuration files for the default server block.
    include /jet/etc/nginx/conf.d/*.inc;
}

I have also attached a screen shot but removed the domain name, I would be really great full for any help. I am unable to sleep until I resolve the issue therefore using the phrase "I am panicking" is an understatement!

On one browser i get the following:

www...... uses an invalid security certificate. The certificate expired on 29 November 2017, 23:59. The current time is 30 November 2017, 03:33.

Which does apply to the old certificate that has been replaced and the new does not expire until February 2018. I have verified this has definitely been removed from the server but Firefox and other browsers except ios seem to think it has not been removed, could this be down to cache somewhere along the lines, If so.. does that mean all my visitors that have been to the site will see this message unless their a unique visitor?

Answer 1

Did you? :

# service stop nginx
# ps auxw|grep nginx

There must be no living nginx any more at this point. If there is the kill it. After that:

# service start nginx

If you stop and start nginx and make sure "stop" really killed the process and /jet/etc/nginx/ssl/cert.crt is really pointing to the new certificate (double check by dumping that cert.crt with openssl, verify that the Validity fields are at the correct date) then there's no way the old certificate could still be used by nginx.

Also, are you sure it's nginx picking up the connection? I.e. if you do a host www.your.site and connect to that IP with ssh and do a netstat -npa|grep 443 there, then the process that you get really is that nginx and not some other proxy (varnish, apache, haproxy, docker picking up the connection etc.)?

If in fact it is nginx that is picking up the connection, then you have to see that connection getting served by nginx in its logs /var/log/nginx/*.log.

If you have a more complex nginx setup it might be that there's multiple configuration items setting up the certificate. So do a grep -r ssl_certificate /etc/nginx. Is the certificate set in one place only or are there multiple sites setting it? Is another ssl_certificate setting being used instead of the one you want?

Also make sure it's not low level packet rewriting or such that is moving the connection somewhere else: check iptables -L and iptables -t NAT -L.

Answer 2

Did you add the Comodo CA bundle to your certificate? You should have received it or a link to it with the certificate. It needs to be added to the certificate file after the site certificate to work with nginx. To test your certificate, use the Qualys SSL Test.

Answer 3

I see you already found an answer, but I'll add my own experience in case it might help others.

I had a similar problem after installing a custom certificate from DigiCert that was generated without without an issuer reference field for an internal domain. Instead, DigiCert provided an issuer certificate that clients can install in their local keychain. Chrome and Firefox (and probably other browsers I haven't checked) will use your local keychain and also follow URI's until they find an issuer certificate signed by a trusted certificate authority if it cannot find the issuer certificate in the browser's pre-configured collection. This is known as following a "chain of trust".

The python requests library (based on urllib3), on the other hand, won't follow a chain of trust due to security concerns. It will use only a preconfigured set of certificates provided by the python-certifi project. When I wrote a python script to access this web service, I got a similar error message to the one you described. The certificate was valid. urllib3 just wouldn't verify it, even after I installed it in my host keychain.

In order to proceed, I wrote the cert-session Python package. You can read more about it in this article.

So it sounds like nothing was wrong with your nginx configuration. Clients on your iPhone and iPad could verify the certificate because they follow the issuer links until they find a trusted certificate authority. But you have clients on other devices that won't follow the chain of trust. Hence, they wouldn't verify the certificate.

It probably got resolved when you ran a setup script that installed the issuer certificate locally. That's my educated guess.