0

Recently the servers hosting our puppet server went down.

After redeploying the containers there seems to be an SSL issue.

2018-01-16T14:36:49.770274413Z Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem

2018-01-16T14:36:49.770278010Z  at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)

2018-01-16T14:36:49.770281700Z  at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)

2018-01-16T14:36:49.770285230Z  at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)

2018-01-16T14:36:49.770288860Z  at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)

2018-01-16T14:36:49.770292535Z  at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)

2018-01-16T14:36:49.770296037Z  at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)

2018-01-16T14:36:49.770299517Z  at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)

2018-01-16T14:36:49.770303285Z  at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)

2018-01-16T14:36:49.770306850Z  at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)

2018-01-16T14:36:49.770310430Z  at java.security.AccessController.doPrivileged(Native Method)

2018-01-16T14:36:49.770314068Z  at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)

2018-01-16T14:36:49.770317603Z  at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)

2018-01-16T14:36:49.770321175Z  at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)

2018-01-16T14:36:49.770324797Z  ... 9 common frames omitted

2018-01-16T14:36:49.770328925Z Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed

2018-01-16T14:36:49.770336317Z  at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:352)

2018-01-16T14:36:49.770340178Z  at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:260)

2018-01-16T14:36:49.770344615Z  at sun.security.validator.Validator.validate(Validator.java:260)

2018-01-16T14:36:49.770350867Z  at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)

2018-01-16T14:36:49.770355767Z  at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)

2018-01-16T14:36:49.770359543Z  at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)

2018-01-16T14:36:49.770363103Z  at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501)

2018-01-16T14:36:49.770366760Z  ... 17 common frames omitted

2018-01-16T14:36:49.770370253Z Caused by: java.security.cert.CertPathValidatorException: timestamp check failed

2018-01-16T14:36:49.770373823Z  at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)

2018-01-16T14:36:49.770377522Z  at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:219)

2018-01-16T14:36:49.770381140Z  at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140)

2018-01-16T14:36:49.770384758Z  at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79)

2018-01-16T14:36:49.770388458Z  at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)

2018-01-16T14:36:49.770392038Z  at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:347)

2018-01-16T14:36:49.770395575Z  ... 23 common frames omitted

2018-01-16T14:36:49.770399060Z Caused by: java.security.cert.CertificateExpiredException: NotAfter: Mon Jan 15 18:12:18 UTC 2018

2018-01-16T14:36:49.770402708Z  at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:274)

2018-01-16T14:36:49.770408587Z  at sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:629)

2018-01-16T14:36:49.770413647Z  at sun.security.provider.certpath.BasicChecker.verifyTimestamp(BasicChecker.java:190)

2018-01-16T14:36:49.770419840Z  at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144)

2018-01-16T14:36:49.770429403Z  at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)

2018-01-16T14:36:49.770443412Z  ... 28 common frames omitted

2018-01-16T14:36:49.774570269Z 2018-01-16 14:36:49,774 WARN  [puppetserver] Puppet Error connecting to puppetdb on 8081 at route /pdb/cmd/v1?checksum=6a40b1127a0e8c1dee4fdd40cd45c9a9b4478dc6&version=8&certname=2klic-dev-596e89d2fe5e08410003f2e6&command=store_report&producer-timestamp=1516113409, error message received was 'Error executing http request'. Failing over to the next PuppetDB server_url in the 'server_urls' list

2018-01-16T14:36:49.776385101Z 2018-01-16 14:36:49,776 ERROR [puppetserver] Puppet Failed to execute '/pdb/cmd/v1?checksum=6a40b1127a0e8c1dee4fdd40cd45c9a9b4478dc6&version=8&certname=2klic-dev-596e89d2fe5e08410003f2e6&command=store_report&producer-timestamp=1516113409' on at least 1 of the following 'server_urls': https://puppetdb:8081

2018-01-16T14:36:49.777516859Z 74.57.127.213 - - - 16/Jan/2018:14:36:49 +0000 "PUT /puppet/v3/report/2klic-dev-596e89d2fe5e08410003f2e6?environment=2klic_smart_controller_ws1_2_beta& HTTP/1.1" 200 12 74.57.127.213 74.57.127.213 8140 246

I removed puppetdb as a node and had it check back in and the SSL process went fine. But PuppetDB still hasn't been recording data from the check-ins.

I also looked at the ca file on PuppetDb to check if it was expired:

openssl x509 -enddate -noout -in /etc/puppetlabs/puppetdb/ssl/ca

output notAfter=Jun 25 20:16:09 2022 GMT.

Similar output on puppet server ca:

openssl x509 -enddate -noout -in /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem

Project structure based on: https://github.com/puppetlabs/puppet-in-docker-examples/blob/master/compose/docker-compose.yml

  • maybe this helps: stackoverflow.com/questions/3670503/… check if the dates inside the containers are OK. check if the correct keystore and therefor the correct key is actually used, and not some older one you might have created previously which did actually run out. this looks like either a time issue, or possibly even a wrong used keystore, though i have no further info on the details. – Dennis Nolte Jan 19 '18 at 14:06
  • The client truststore seems to be missing the needed certificate to handshake. You can test manually using these instructions - confluence.atlassian.com/kb/… – Schrute Jan 23 '18 at 20:35
0

The first thing I'd check is that you have time synchronized on all your servers via ntp/chrony since the stack trace stated: "timestamp check failed".

It also might be helpful to see what openssl s_client returns about errors when trying to make a connection:

openssl s_client -connect YOUR-PUPPET-DB-HOST:8081
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.