Connection to URL using OpenSSL client works, but curl fails

Question

I have a CentOS 5.9 server, from which I need to make SSL connections to another server. The remote server has a certificate ultimately signed by the GeoTrust Global CA. At the time of writing, this certificate is the second one listed on GeoTrust's download page. I am getting inconsistent results, depending on whether I use OpenSSL or curl to make the connection:

openssl s_client -connect <server>:443 -CAfile /path/to/GeoTrustCA.pem

works fine, but

curl --cacert /path/to/GeoTrustCA.pem https://<server>/

fails with the standard "could not verify certificate" error.

Here are details of the tools I'm using:

$ curl --version 
curl 7.15.5 (i386-redhat-linux-gnu) libcurl/7.15.5
OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 Protocols: tftp ftp telnet dict
ldap http file https ftps  Features: GSS-Negotiate IDN IPv6 Largefile
NTLM SSL libz

and

$ openssl version
OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008

I'm stumped. The server I'm connecting to has been working with no apparent problems for years: I've never heard of anyone or any system being unable to connect to it like this.

Answer 1

I was curious about this so I did some testing. From what I can tell, there appears to be a fundamental difference in what openssl and curl take into consideration when using their root CA switches (-CAfile and --cacert respectively).

When the --cacert switch is used in curl, it appears to ONLY use the admin-specified root during verification. For example, I download the GeoTrust PEM file you mentioned earlier and tried using it to fetch a page from yahoo:

[foo@foobox tmp]# curl --cacert /tmp/geotest.pem https://info.yahoo.com/
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). The default
 bundle is named curl-ca-bundle.crt; you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Now, I tried the same test with openssl:

[foo@foobox tmp]# openssl s_client -connect info.yahoo.com:443 -CAfile /tmp/geotest.pem
CONNECTED(00000003)
depth=3 /C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
verify return:1
depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
verify return:1
depth=1 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
verify return:1
depth=0 /C=US/ST=California/L=Sunnyvale/O=Yahoo Inc./CN=www.yahoo.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/L=Sunnyvale/O=Yahoo Inc./CN=www.yahoo.com
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
 1 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
 2 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
...
...
...
...
...
...
SSL handshake has read 5342 bytes and written 435 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA
    Session-ID: 6CAE87314ED66784B25C0FB36197D822CC73032FBFF30AD9E37CFF3D1678EBCC
    Session-ID-ctx:
    Master-Key: 6B9135F16512A251AB6DBEF62C6B261EC31DB90A0076C33DD67B27EAAB83A0333D50B1B7F10727DE47AB051A9C3A0499
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1415842989
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

You'll note that there are no errors in the initial chain description and that it shows Verify return code: 0 (ok) at the bottom.

So even though GeoTrust isn't mentioned at all in the certificate chain, OpenSSL is somehow able to validate/verify the root.

Hmmm...

That got me to thinking about where the default truststore for openssl is... for some reason I had difficulty finding information about it online (I'm sure it's well documented and I'm just blind). Upon poking around on my system, I came across /etc/pki/tls/certs/ca-bundle.crt

I searched in ca-bundle.crt and removed the root that yahoo uses (/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority) and ran the same exact command again:

[foo@foobox tmp]# openssl s_client -connect info.yahoo.com:443 -CAfile /tmp/geotest.pem
CONNECTED(00000003)
depth=2 /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=US/ST=California/L=Sunnyvale/O=Yahoo Inc./CN=www.yahoo.com
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
 1 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)10/CN=VeriSign Class 3 Secure Server CA - G3
   i:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
 2 s:/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=(c) 2006 VeriSign, Inc. - For authorized use only/CN=VeriSign Class 3 Public Primary Certification Authority - G5
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
...
...
...
...
...
...
...
...
---
SSL handshake has read 5342 bytes and written 435 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA
    Session-ID: 09D998B153574D5C785BFF191B99CAB8BFCEF4DAC482F75A601886E668BF9CE6
    Session-ID-ctx:
    Master-Key: 1F98289FEB5926B8814D5E3B163FB40CC03BBC5C2D8A0045C0DFF0532458F18F722D5FD53155327B0A78627E3FE909E5
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1415843859
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---

This time, we get verification errors.

Soooooo, with all that said, I'm inclined to suspect the following:

• The root certificate you're using may not actually match the root that you need to use.
• curl ONLY uses the root you tell it to use.
• openssl uses the root you point it to ANNNND whatever roots are in its default truststore.

As for why openssl does this? No idea. The documentation for that switch doesn't mention this workflow/behavior:

-CAfile file

A file containing trusted certificates to use during server authentication and to use when attempting to build the client certificate chain.

Perhaps someone else can examine the code for openssl and elaborate further.