17

If I try to access our HTTPS server that has certbot-issued certificate from debian 9, I get the following error:

 # curl -v https://hu.dbpedia.org/
 *   Trying 195.111.2.82...
 * TCP_NODELAY set
 * Connected to hu.dbpedia.org (195.111.2.82) port 443      (#0)
 * ALPN, offering h2
 * ALPN, offering http/1.1
 * Cipher selection:      ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
 * successfully set certificate verify locations:
 *   CAfile: /etc/ssl/certs/ca-certificates.crt
   CApath: /etc/ssl/certs
 * TLSv1.2 (OUT), TLS header, Certificate Status (22):
 * TLSv1.2 (OUT), TLS handshake, Client hello (1):
 * TLSv1.2 (IN), TLS handshake, Server hello (2):
 * TLSv1.2 (IN), TLS handshake, Certificate (11):
 * TLSv1.2 (OUT), TLS alert, Server hello (2):
 * SSL certificate problem: certificate has expired
 * Curl_http_done: called premature == 1
 * stopped the pause stream!
 * Closing connection 0
 curl: (60) SSL certificate problem: certificate has expired

However, if I try the same command from debian 10, it succeeds.

I tried to simply copy all ca-certificates from a debian 10 VM to the debian 9 VM (to /usr/local/share/ca-certificates) with rsync, and then ran update-ca-certificates which seemingly added 400+ certificates. Unfortunately, it did not help. This is no wonder, as it seems that there are the same certificates on both debian 9 and 10, apparently.

My question is: How can I access sites with certbot certificates from debian 9 machines without ignoring certificate verification altogether

2
  • @GeraldSchneider Unfortunately, that is not an option, at least in the short term. I would like to have a temporary solution until the upgrade can be made.
    – P.Péter
    Sep 30 '21 at 16:56
  • Fun fact: old (~4 year old) versions of OSX suffer from the same problem. They also probably use and old openssl version.
    – P.Péter
    Oct 6 '21 at 8:18
23

First off, Debian 9 is EOL. But as the clients may not be under your control, you may of course want to try to cater to them in this breakage.

I assume that while the question only mentions certbot, that it's really specifically about Letsencrypt.
(The tool certbot itself is an ACME protocol client which is also used with other ACME-based CAs, so there is some room for confusion here.)

The problem at hand would appear to be the combination of:

  • The old Letsencrypt root ("DST Root CA X3") has expired
  • The new default LE chain tries to be "extra compatible" by presenting an optional extension of the chain where the new root ("ISRG Root X1") is presented as a cross-signed intermediate for the old root (as very old Android versions still accept the expired root, but do not have the new root)
  • Openssl 1.0 has a bug causing it to just try the first chain it sees and if it doesn't like that, it doesn't look at any other possibilities (ie, the shorter new chain ending at X1 vs the longer "compatibility" extension of that chain going through X1 to X3).
  • libcurl3 on Debian 9 is linked to libssl 1.0

If you instead present the new LE certificate chain that does not try to be extra compatible, just ending at the new root (X1), it allows libssl 1.0 to work (but then you lose compatibility with really old Android instead).

Other than that, other CAs (ACME or otherwise) is probably the option to consider.

6
  • 1
    This was the best explanation for the problem, thank you!
    – P.Péter
    Oct 1 '21 at 6:43
  • 4
    By the way, the most "debian" way of disabling a certificate seems to be running dpkg-reconfigure ca-certificates and deselecting the offending one from the list.
    – P.Péter
    Oct 1 '21 at 6:49
  • 3
    Debian 9 is not really End-Of-Life, Debian 8 reached EOL this summer but Debian 9 continues to be supported with security updates.
    – Orphans
    Oct 4 '21 at 11:44
  • It is a great explanation of the cause of the problem, but is it possible to fix it without upgrading to Debian 10 ?
    – bN_
    Oct 26 '21 at 11:47
  • 1
    Edit: @P.Péter using dpkg-reconfigure ca-certificates to deselect "DST Root CA X3" solves the problem. The accepted answer actually provides explanations but no solution. It should be updated with @P.Péter comment.
    – bN_
    Oct 26 '21 at 11:56
20

Warning! Please plan OS upgrade path. The below advice should be applied only in emergency situation to quickly fix a critical system.

Below solution works on Debian Jessy 8 and should on Stretch 9 missing updates too. I just tested it on docker run -it debian:jessie bash, apt-get update && apt-get install curl.

Before:

# curl -I https://hu.dbpedia.org
curl: (60) SSL certificate problem: certificate has expired
More details here: http://curl.haxx.se/docs/sslcerts.html
....

First check if you have offending DST Root CA X3 cert present:

# grep X3 /etc/ca-certificates.conf 
mozilla/DST_Root_CA_X3.crt

Old Debian releases have the proper ISRG Root X1 present too:

# grep X1 /etc/ca-certificates.conf 
mozilla/ISRG_Root_X1.crt

This is going to disable X3:

# sed -i '/^mozilla\/DST_Root_CA_X3/s/^/!/' /etc/ca-certificates.conf && update-ca-certificates -f

Your domain responds to curl fine:

# curl -I https://hu.dbpedia.org/
HTTP/1.1 200 OK
...

Again, plan an upgrade please.

3
  • This removing/commenting out the cert in the conf file worked for us as well. Thank you for the help.
    – Ray
    Oct 1 '21 at 16:29
  • I can confirm that this works on Debian Stretch too.
    – Dominik
    Oct 7 '21 at 7:36
  • Works on three of my Debian GNU/Linux 10 (buster) servers. Four of them went down in 1 day and I had to strip them mostly of curl for them to work. One website hosted on Debian Buster isn't working still, but it may be a different error I suppose as the previous three websites went down on October 1, but this one with the remaining error went down January 4ish. Thank you!
    – luminol
    Jan 13 at 20:36
5

DST Root CA X3 root certificate expired on Sep 30 14:01:15 2021 GMT. It was used as one of certification paths for Let’s Encrypt certificates Older cURL version has a bug that will cause expired root to fail connection instead of trying other roots in local ca store.

As of 30 september 2021 ca store for curl (https://curl.se/docs/caextract.html) still contains expired DST Root CA X3 root certificate so updating it won't fix problem. You can either update your cURL (which might be quite challenging in some situations) or edit local ca store (f.e. /etc/pki/tls/certs/ca-bundle.crt) and manually remove certificate after line "DST Root CA X3"

2

I've just had this problem with a Docker image based on Debian Stretch. The solution has been easy, I suppose Debian has patched libssl1.0.2 to fix the chain selection issue Håkan has mentioned. In my case, updating just the certificates and libssl1.0.2 by running apt install -y libssl1.0.2 ca-certificates (with apt update for containers beforehand) has been enough.

2

For Debian 8 and 9, I just do that for updating cert on host and no longer have the error 60: SSL certificate problem: certificate has expired : https://github.com/xenetis/letsencrypt-expiration

Just run, it should work :

wget -O - https://raw.githubusercontent.com/xenetis/letsencrypt-expiration/main/letsencrypt-expiration.sh | bash
1
  • Awesome, thank you! I will move on from Debian 9 eventually...
    – jakethedog
    Nov 12 '21 at 12:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.