I'm slowly transitioning from an exclusive developer role and into more of a hybrid DevOps role at my company. Which means I'm new to a lot of this, please go easy on me... :-p

My client's server is running Ubuntu 16.04, with PHP 5.6.4 and there is a function in their site's administrative portal that runs a curl command (essentially) back to itself for some sort of file syncing. And it's been failing for some time (a few weeks/months). The problem (I think) is that certificate verification is failing, and thus, the function is dying on the vine.

When I ssh into the server, I can easily curl out to anywhere with no issues (Google, example.org, etc...). But trying to just a basic curl to the site's main url borks.

$ curl -v https://www.[my-site-name].com

*   Trying [my-site-IP]...
* Connected to [my-site-name] ([my-site-IP]) port 443 (#0)
* found 258 certificates in /etc/ssl/certs/ca-certificates.crt
* found 908 certificates in /etc/ssl/certs/
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
* Closing connection 0
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

I know that I can run curl with -k for it to be insecure, but I'm hesitant to do so. I guess the first question is, should I not worry about running this with the insecure flag since it's technically not leaving the server at all? I've tested this exact same curl command on one of our newer boxes running Ubuntu 18.x and also on a DigitalOcean running v20 with no issues at all -- both external and internal curls worked great.

I can even be on another server, and curl back to my one experiencing issues, and that is working fine too.

I've tried everything I could think of (which admittedly isn't much) and nothing seems to be working.

  • run updates for curl and certbot packages
  • forcing the update-ca-certificates
  • added /etc/ssl/certs/cacert.pem to both the curl.cainfo and openssl.cafile vars in php.ini

I know this probably doesn't matter, but just for completeness, I've also run the site through various online verification services:

All came back with positive results. The only negative (I guess) is that SSLLabs graded us with a 'B' because apparently TLS 1.0 is still enabled.

Any help would be greatly appreciated. I feel like reading the docs mentioned in the failure warning isn't really all that helpful.

Suggestions / tips / tricks ??

1,000 thank you's in advance!

  • 2
    Update the server. Ubuntu 16.04 is end of life and openssl and everything else related to encryption is so old that is just not usable anymore. Update the server to a supported release. Feb 3, 2022 at 21:24
  • @GeraldSchneider Thanks, and I agree!! That is exactly what I had proposed to our PM in our scrum this morning. But... it's (unfortunately) not in the cards right now. To make matters worse... they are moving to Pantheon hosting in... August (?!?), but short term need to have this fixed on the current box.
    – Drew
    Feb 3, 2022 at 21:37
  • The versions of curl and certbot are irrelevant. Try updating the ca-certificates package; launchpad claims 20210119~16.04.1 is current. Note the command update-ca-certificates has nothing to do with updating the package, but rather making site or manual overrides to the data from the package. If that doesn't help, you'll have to compare the cert(s) being used by the server to the truststore, which will require actual learning and thinking. Feb 4, 2022 at 1:41
  • Thanks @dave_thompson_085. Looks like I already had the latest version of the ca-certificates package. And... hah, I totally agree about the "actual learning and thinking" part of the comment above. Coming from a design background, then moving into development, and now -slowly- getting into the DevOps/SysAdmin role... it's been a little daunting and confusing, for me at least.
    – Drew
    Feb 4, 2022 at 14:57
  • Okay, getting harder. See semi-answer. Feb 5, 2022 at 12:04

1 Answer 1


Meta: not an answer, at least not yet, but some info and advice.

Okay, easy case doesn't apply. As you've encountered, curl doesn't give very helpful information when it gets a cert-verify error from the underlying SSL/TLS stack, at least when using GnuTLS as the Ubuntu build does. However, OpenSSL (which should be present because curl has it as a dependency, though I don't see why) provides a commandline program named simply openssl that is mostly a test tool, and is good deal more verbose as well as more error-tolerant. For example, on an intentionally bad site, in a docker version of Ubuntu 16.04:

# curl -v https://untrusted-root.badssl.com/
*   Trying
* Connected to untrusted-root.badssl.com ( port 443 (#0)
* found 129 certificates in /etc/ssl/certs/ca-certificates.crt
* found 516 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
* Closing connection 0
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html

[snip canned text -- same as yours]
# openssl s_client -connect untrusted-root.badssl.com:443 -servername untrusted-root.badssl.com
depth=1 C = US, ST = California, L = San Francisco, O = BadSSL, CN = BadSSL Untrusted Root Certificate Authority
verify error:num=19:self signed certificate in certificate chain
Certificate chain
 0 s:/C=US/ST=California/L=San Francisco/O=BadSSL/CN=*.badssl.com
   i:/C=US/ST=California/L=San Francisco/O=BadSSL/CN=BadSSL Untrusted Root Certificate Authority
 1 s:/C=US/ST=California/L=San Francisco/O=BadSSL/CN=BadSSL Untrusted Root Certificate Authority
   i:/C=US/ST=California/L=San Francisco/O=BadSSL/CN=BadSSL Untrusted Root Certificate Authority
Server certificate
subject=/C=US/ST=California/L=San Francisco/O=BadSSL/CN=*.badssl.com
issuer=/C=US/ST=California/L=San Francisco/O=BadSSL/CN=BadSSL Untrusted Root Certificate Authority
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
SSL handshake has read 3561 bytes and written 425 bytes
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: AC5EB655FAEA1C1A320CA930E6A087FBFE020D3D9C451A72DD7CE77A8080AF0D
    Master-Key: 419065F69901D74454A224DD22F9D459FA85E6EB4D6F043C649C8FDFA2E5E1FD9C168AC270087B88511B0939927F9A0A
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - cc 91 34 52 b1 ce c6 7c-8a 97 27 b0 b4 50 d6 9d   ..4R...|..'..P..
    0010 - 27 b8 87 13 0b 70 92 e7-23 be 57 5c 00 31 f7 90   '....p..#.W\.1..
    0020 - 65 e3 08 d5 63 14 e9 cd-cc 50 59 3f 2a 98 31 d1   e...c....PY?*.1.
    0030 - ab f8 ca a8 09 aa 66 bf-7c ff b6 66 42 10 8b 6e   ......f.|..fB..n
    0040 - c7 e8 7b f3 2b 35 b3 76-8c 95 b3 b5 37 85 09 42   ..{.+5.v....7..B
    0050 - 83 a9 c3 f7 54 f4 c5 f5-b2 0d d5 fa c6 24 a6 e2   ....T........$..
    0060 - fb 03 cf 35 9c 0d cc 48-e0 bc fc 43 11 3c 19 51   ...5...H...C.<.Q
    0070 - 0a 23 7d b2 6c ff 9b e2-bc 64 1d 74 34 cb 13 7b   .#}.l....d.t4..{
    0080 - c8 55 47 95 71 9d 94 00-33 a0 a0 87 d4 4a fb 81   .UG.q...3....J..
    0090 - 34 36 28 2e ac d7 22 36-36 5f 9c cb 3f 0d e7 00   46(..."66_..?...
    00a0 - e2 b7 d0 35 48 81 2f c7-9d a1 8a f6 52 a3 11 17   ...5H./.....R...
    00b0 - 67 89 94 d3 66 2b 5c c4-73 c5 72 1e 84 46 57 a5   g...f+\.s.r..FW.
    00c0 - 77 05 bc 49 a3 1a fe e6-da a8 0f 40 e2 17 f0 40   w..I.......@...@

    Start Time: 1644060097
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)

Observe that the first output after the CONNECTED line: depth=1 ... / verify error:num=19:.... indicates the cert verification failed; however openssl ignores this error and completes the handshake, resulting in the last section of output: SSL-Session: / Protocol: (valid) / Cipher: (valid) / ... / Verify return code: ... after which you must type Q and return or control-D (alone) to exit the program, or control-C to kill, unless you add </dev/null on the commandline which has the same effect as automatically typing control-D. For now you don't need to understand the specific values of Protocol and Cipher, just that they are NOT something like 'none' or 'error' or 'missing'. But be warned that OpenSSL's message text for verify=19 is incomplete to the point of being confusing; it is perfectly normal in general for the cert chain from an SSL/TLS server to include the root cert which is self-signed, and if that root is in the local truststore OpenSSL does (and curl/GnuTLS certainly should) accept such a chain. It is only if the root is in the chain AND NOT locally trusted that you get this verify code. Other verify codes like 'expired' have mostly clearer messages.

Also note I used -servername with the host name; this invokes SNI=ServerNameIndication which many SSL/TLS servers today, including badssl.com, need in order to serve the correct certificate(s). However, the version of curl in Ubuntu16.04 does NOT send SNI; without research, I don't know if this is due to the slightly (but not very) older version of curl, the use of GnuTLS, or just a mistake in the build. But anyway as a test try the openssl s_client command against your server both with and without the -servername host part, and save the output (with redirection or tee, preferably including stderr with &> or |& tee or equivalent) because we may need them. If the with-SNI version works and the without-SNI version gets a verify error, that's your problem -- lack of SNI. (It may be difficult to fix, but at least you know what it is.)

Otherwise, look at the output for without-SNI. If OpenSSL reports a verify error (agreeing with curl/GnuTLS), look at the related message -- it should at least point in the direction of the problem, if not exactly to it. If OpenSSL reports no verify error (where curl/GnuTLS does, both using the same system-supplied truststore in /etc/ssl) it's going to be more complicated, so I'll leave that for now and add later if needed.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .