I'm diagnosing a TLS certificate verification problem in Ubuntu Xenial that resulted from the recent Let's Encrypt root certificate expiration. The problem happens in cURL which uses GnuTLS under the hood -- so I've used gnutls-cli to diagnose it (when using openssl, a different problem happens even before certificate verification -- so it's of no use).

Despite the fact that I've updated the DST Root CA X3 certificate (with apt install ca-certificates) and confirmed that by comparing the contents of /usr/share/ca-certificates/mozilla/ (by file names, sizes and MD5 sums) on the problem machine and on another one where there's no error, the problem persists. Diffing the contents of /etc/ssl/certs/ca-certificates.crt (which is the file that GnuTLS directly uses, allegedly autogenerated from the former directory by /usr/sbin/update-ca-certificates) does show differences, but the DST Root CA X3 certificate is not among them.

Now I'm stuck at the point where I can reproduce the problem with gnutls-cli -- but it doesn't show which root certificate it uses, only the server-provided certificate chain which is all okay:

$ gnutls-cli download.clis.cloud.ibm.com --print-cert </dev/null 2>&1
Processed 129 CA certificate(s).
Resolving 'download.clis.cloud.ibm.com'...
Connecting to ''...
- Certificate type: X.509
- Got a certificate list of 3 certificates.
- Certificate[0] info:
 - subject `CN=cert-00045-cdnedge-bluemix.akamaized.net', issuer `C=US,O=Let's Encrypt,CN=R3', RSA key 2048 bits, signed using RSA-SHA256, activated `2021-10-19 02:05:23 UTC', expires `2022-01-17 02:05:22 UTC', SHA-1 fingerprint `8a13f222870579984cf6a0c6cf8ebfe6f122eb0b'
    Public Key ID:
    Public key's random art:
- Certificate[1] info:
 - subject `C=US,O=Let's Encrypt,CN=R3', issuer `C=US,O=Internet Security Research Group,CN=ISRG Root X1', RSA key 2048 bits, signed using RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15 16:00:00 UTC', SHA-1 fingerprint `a053375bfe84e8b748782c7cee15827a6af5a405'
AYYwHQYDVR*** Fatal error: Error in the certificate.
*** Handshake has failed
GnuTLS error: Error in the certificate.
- Certificate[2] info:
 - subject `C=US,O=Internet Security Research Group,CN=ISRG Root X1', issuer `O=Digital Signature Trust Co.,CN=DST Root CA X3', RSA key 4096 bits, signed using RSA-SHA256, activated `2021-01-20 19:14:03 UTC', expires `2024-09-30 18:14:03 UTC', SHA-1 fingerprint `933c6ddee95c9c41a40f9f50493d82be03ad87bf'
- Status: The certificate is NOT trusted. The certificate chain uses expired certificate. 
*** PKI verification of server certificate failed...

So I cannot see which local certificate keeps causing the problem. Is there a way to retrieve this information?

1 Answer 1


The problem you're running into is that your gnutl-cli command is printing out the details of the TLS connection to the server. As for the certs themselves, it just dumps the PEM encoded cert and then moves on. So when you run gnutls-cli download.clis.cloud.ibm.com --print-cert </dev/null 2>&1, it shows you a bunch of info and then three blocks of PEM encoded cert info, seen between the "-----BEGIN CERTIFICATE-----" line and the "-----END CERTIFICATE-----" line. The stuff between those lines is the actual certs. The first cert is your local certificate. The second one is an intermediate certificate. The third one is the root certificate. You need to capture the cert info from that third one and view it with something that's able to decode those blobs of text.

You can manually copy the cert info (the stuff between "-----BEGIN CERTIFICATE-----" line and the "-----END CERTIFICATE-----" line, inclusive) and paste it somewhere, then view it using the openssl x509 command. Or you can just take the output of gnutls-cli and feed it to openssl directly. If you do that, you'll have to capture just the third cert (the root cert that you want to see) and send it to openssl. I used awk to do that. Altogether, this works:

$ gnutls-cli download.clis.cloud.ibm.com --print-cert </dev/null 2>&1 | awk '/Certificate\[2\]/,0' | openssl x509 -noout -text
        Version: 3 (0x2)
        Serial Number:
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
            Not Before: Jan 20 19:14:03 2021 GMT
            Not After : Sep 30 18:14:03 2024 GMT
        Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
<...and so on>

So that takes your command as you had it, pipes it to awk, which searches for a line with "Certificate[2]" and everything after it to the end of the file/text/stream, and pipes that on to the openssl x509 command. The openssl command says "show me this x509 certificate in human readable text format". The Issuer CN, when I ran this just now, is "DST Root CA X3".

Note also: the openssl command is smart enough pull out only the PEM encoded certificate info from all the text that you pass on to it. But if you don't use awk to filter out the previous cert info, openssl is gonna grab the first block it sees (the local installed cert) and show just it. That's why you have to filter the output of gnutls-cli to pass on a blob of text with only the third cert in it.

  • The final certificate says: subject "C=US,O=Internet Security Research Group,CN=ISRG Root X1", issuer "O=Digital Signature Trust Co.,CN=DST Root CA X3". This is not a root certificate: in a root certificate, the subject and issuer would be the same. That's what makes me think that the locally-stored root certificate being used is not printed by gnutls-cli at all. Jan 22, 2023 at 11:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .