Curl: unable to get local issuer certificate. How to debug?

Question

I’ve got an odd problem. Updated my LAMP dev machine (Debian) to PHP 7. Afterwards I cannot connect to a specific TLS encrypted API via Curl anymore.

The SSL cert in question is signed by thawte.

curl https://example.com

gives me

curl: (60) SSL certificate problem: unable to get local issuer certificate

whereas

curl https://thawte.com

which—of course—is also signed by Thawte works.

I can access the API site via HTTPS on other machines, e.g. my Desktop via curl and in the browser. So the cert is definitly valid. SSL Labs rating is A.

Any other Curl requests from my dev machine to other SSL encrypted sites work. My root certs are up to date. To verify, I ran update-ca-certificates. I even downloaded http://curl.haxx.se/ca/cacert.pem to /etc/ssl/certs and ran c_rehash.

Still the same error.

Is there any way to debug the verifcation process and see which local issuer certificate curl (or openssl) is looking for but not finding, i.e. a file name?

UPDATE

curl -vs https://example.com

tells me (IP+Domain anonymized)

* Hostname was NOT found in DNS cache
*   Trying 192.0.2.1...
* Connected to example.com (192.0.2.1) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0

And

echo | openssl s_client -connect example.com:443

gives

CONNECTED(00000003)
depth=2 C = US, O = "thawte, Inc.", OU = Certification Services Division, OU = "(c) 2006 thawte, Inc. - For authorized use only", CN = thawte Primary Root CA
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=DE/ST=XYZ/CN=*.example.com
   i:/C=US/O=thawte, Inc./CN=thawte SSL CA - G2
 1 s:/C=US/O=thawte, Inc./CN=thawte SSL CA - G2
   i:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA
 2 s:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA
   i:/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/CN=Thawte Premium Server CA/[email protected]
---
Server certificate
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
subject=/C=DE/ST=XYZ/CN=*.example.com
issuer=/C=US/O=thawte, Inc./CN=thawte SSL CA - G2
---
No client certificate CA names sent
---
SSL handshake has read 4214 bytes and written 421 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: [...]
    Session-ID-ctx:
    Master-Key: [...]
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 5a 95 df 40 2c c9 6b d5-4a 50 75 c5 a3 80 0a 2d   Z..@,.k.JPu....-
    [...]
    00b0 - d5 b9 e8 25 00 c5 c7 da-ce 73 fb f2 c5 46 c4 24   ...%.....s...F.$

    Start Time: 1455111516
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---
DONE

Answer 1

Using openssl s_client -connect thawte.com:443 shows:

---
Certificate chain
 0 s:/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Delaware/O=Thawte, Inc./C=US/ST=California/L=Mountain View/businessCategory=Private Organization/serialNumber=3898261/OU=Infrastructure Operations/CN=www.thawte.com
   i:/C=US/O=thawte, Inc./CN=thawte Extended Validation SHA256 SSL CA
 1 s:/C=US/O=thawte, Inc./CN=thawte Extended Validation SHA256 SSL CA
   i:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2008 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA - G3
---

That last "i" shows the issuing self-signed root CA. I'm guessing that that particular Thawte root CA, _i.e. the Primary Root CA - G3 cert, is not in your /etc/ssl/certs directory (as stated in the curl output; openssl s_client does not have a default CA path, and needs to given one explicitly, e.g. -CApath /etc/ssl/certs).

Adding that certificate explicitly to your /etc/ssl/certs directory (and re-running c_rehash) certainly wouldn't hurt. And if it works, e.g. as verified using openssl s_client -connect example.com:443 -CApath /etc/ssl/certs, then you know that that update-ca-certificates command may need some examination/debugging, as to why it hadn't picked up this root CA.

Now, it may be that the above root CA is already in your /etc/ssl/certs directory, and the above steps had no effect. In that case, there are two other issuing CA certs to check (at least in the cert chain offered by thawte.com:443): thawte Primary Root CA, and thawte SSL CA - G2. Repeating the above steps to install these certs into your /etc/ssl/certs directory (and re-running c_rehash) might work. Since these two are intermediate CAs, and not root CAs, the absence of one of them would explain your results, and might be expected as overlooked certs by update-ca-certificates.

Hope this helps!

Answer 2

This could be caused by wrong order of site, issuing, intermediate and root certificates in site's public key certificate file.

The browser display certs in reverse top-bottom direction (root, intermediate, issuing, site) but the certificate must be in bottom-top direction (site, issuing, intermediate, root).