curl and wget fail to verify wildcard Let's Encrypt certificate, but browsers work fine

Question

We've obtained a wildcard certificate from Let's Encrypt for civility.social and *.civility.social, using certbot. This works fine on all browsers and with curl and wget when verifying https://civility.social, or https://graphql.civility.social. The A records for those (sub)domains point to the same server, from which the certbot challenge was completed.

Later we've added a subdomain hosted on a different server, meet.. To use the same certificate on a different server, we copied the files making up the wildcard certificate from the original server to meet.. Both servers use NGINX. The problem is that wget and curl fail to fetch anything from meet.civility.social, even though browsers don't complain. wget fails even with --no-check-certificate.

$ $ wget -v --debug --no-check-certificate https://meet.civility.social
Setting --check-certificate (checkcertificate) to 0
Setting --check-certificate (checkcertificate) to 0
DEBUG output created by Wget 1.20.3 on linux-gnu.

Reading HSTS entries from ~/.wget-hsts
URI encoding = ‘UTF-8’
Converted file name 'index.html' (UTF-8) -> 'index.html' (UTF-8)
--2020-06-26 21:39:15--  https://meet.civility.social/
Resolving meet.civility.social (meet.civility.social)... 157.245.170.94
Caching meet.civility.social => 157.245.170.94
Connecting to meet.civility.social (meet.civility.social)|157.245.170.94|:443... connected.
Created socket 3.
Releasing 0x000055ae59be63e0 (new refcount 1).
Initiating SSL handshake.
SSL handshake failed.
Closed fd 3
Unable to establish SSL connection.


$ curl https://meet.civility.social/
curl: (60) SSL certificate problem: unable to get local issuer certificate

What might be going on here?

Answer 1

You need to update ca-certificates package.

On my machine I have no SSL errors:

mvutcovici@DESKTOP-QMNRLV6:~$ curl -v https://graphql.civility.social
*   Trying 167.71.155.126:443...
* TCP_NODELAY set
* Connected to graphql.civility.social (167.71.155.126) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=civility.social
*  start date: Jun  6 08:45:19 2020 GMT
*  expire date: Sep  4 08:45:19 2020 GMT
*  subjectAltName: host "graphql.civility.social" matched cert's "*.civility.social"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x563dffb227c0)
> GET / HTTP/2
> Host: graphql.civility.social
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 500
< server: nginx/1.17.10 (Ubuntu)
< date: Sat, 27 Jun 2020 05:27:14 GMT
< content-type: application/json; charset=utf-8
< content-length: 153
< access-control-allow-origin: *
< etag: W/"99-9KmtOnsArqM8/UCTY4AvEYVjSiM"
< strict-transport-security: max-age=63072000
<
{"errors":[{"message":"Context creation failed: Incorrect method GET from 199.241.131.83 requesting {}","extensions":{"code":"INTERNAL_SERVER_ERROR"}}]}
* Connection #0 to host graphql.civility.social left intact
mvutcovici@DESKTOP-QMNRLV6:~$

Edit 1:

For meet.civility.social the webserver is not sending the intermediate certificate. You can see this with a network capture in WireShark. You are presenting only the certificate which covers *.civility.social and civility.social, but the intermediate certificate Let's Encrypt Authority X3 is omitted. certbot should take care of installing the proper intermediate certificate, for more details see: https://letsencrypt.org/certificates/

You need to concatenate the server and intermediate certificate:

# civility.social
-----BEGIN CERTIFICATE-----
MIIFaTCCBFGgAwIBAgISA4ROk0Mr6HcT5XbuumfDaJrDMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0yMDA2MDYwODQ1MTlaFw0y
MDA5MDQwODQ1MTlaMBoxGDAWBgNVBAMTD2NpdmlsaXR5LnNvY2lhbDCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAMLtBpnuOOOwKUo3j7sTASUcidiE3yVA
LN56oK9wIY5uJ9kqJTRjFDbQrAIfaAv3p6b1KNBEfuBQ6TRbWEuRrFDi4WUCaNQS
UKwK4jIqJhH3fxxZejUjM2hewKbnTgvGpWQyQkk9kDzDW7klo1nffUdj1z0nW14h
z1NxmoMWerxVqfrkuJ9o2Fl7zkvhmr26i+p8ehSJctVQtNnSdeQGGe2eX5D4VQSI
dMmfjzfzl0U9002tjLmY5iUknRf6EIJxrgPb2E81Ay47vsL4FS7AWNvsF3jOomQh
RdP4IjUs3NvmUF7x5YRWFWO9aOabLotc2L4FKiVeLRUGMMgXLwmDy98CAwEAAaOC
AncwggJzMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYB
BQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUpViiA3AsreWjZkSAaOSW+fal
66YwHwYDVR0jBBgwFoAUqEpqYwR93brm0Tm3pkVl7/Oo7KEwbwYIKwYBBQUHAQEE
YzBhMC4GCCsGAQUFBzABhiJodHRwOi8vb2NzcC5pbnQteDMubGV0c2VuY3J5cHQu
b3JnMC8GCCsGAQUFBzAChiNodHRwOi8vY2VydC5pbnQteDMubGV0c2VuY3J5cHQu
b3JnLzAtBgNVHREEJjAkghEqLmNpdmlsaXR5LnNvY2lhbIIPY2l2aWxpdHkuc29j
aWFsMEwGA1UdIARFMEMwCAYGZ4EMAQIBMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYB
BQUHAgEWGmh0dHA6Ly9jcHMubGV0c2VuY3J5cHQub3JnMIIBBAYKKwYBBAHWeQIE
AgSB9QSB8gDwAHYA5xLysDd+GmL7jskMYYTx6ns3y1YdESZb8+DzS/JBVG4AAAFy
iQU30AAABAMARzBFAiAr8zcVbe0ZZD9/hTkvMUe6YE9WE26xqsjhBGE7F4BZ2QIh
ALUtrBQy/FZxIJ+54tvSyS9V3x5DrHEjLYrVSX+XfW6nAHYAB7dcG+V9aP/xsMYd
IxXHuuZXfFeUt2ruvGE6GmnTohwAAAFyiQU4IQAABAMARzBFAiEA/wXl+iyT1h0g
wJT9s/94+MFMf2Vc7mY/WWnXUEs5IgECIC5fmsw4PSpvYjBd/gXGi8HUO7oPSura
OCBR0QBK2iFzMA0GCSqGSIb3DQEBCwUAA4IBAQCILd6E26gp+6YVzhzAmhDHiWi3
d7gg/ttmtCml99Ud5vhWTJFc7ORJVGO2zp4IO2QxBg8aDVIDjIr/QmgUT7+Fdjyi
FXJ1m1KMJIs3gySY1gfBXMN4wSCVmTMlXds1ukGrwrR86ZshmBhIHe/qOnt+o5nT
Y4DGWpWffEp4eNk5EzVAgnXpLFva4JpDJgmtSwBOzIHQEDieqP5JsApRAgt+WYLv
BQwYQsKya/YLCHTzp5MVOjVWpGo+jHSFQEPaxR8MiFfMNSO3+Ymxc4X1w0kHvxFt
aXqTubdI9BoqEcB7d0qnApY2Bw7hkSOgzsgy4r4dvrasLTW+xjAB7S2iXbfP
-----END CERTIFICATE-----
# Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----