1

Summary

I had to rebuild a server, and run into an interesting issue. Puppet refuses to verify the certificates, even after removing /var/lib/puppet/ssl and cleaning the certificate off of the master.

Servers

Master:
OS: Ubuntu 14.04
Puppet Master version: 3.4.3-1 (using Webrick)

Agent:
OS: Ubuntu 12.04
Puppet version: 2.7.11
(Note that I replaced the hostname with 'agent-server.com' in the below output)

Replication Steps

  • Remove SSL dir on agent-server:

    rm -fr /var/lib/puppet/ssl

  • Clean the certificate on the Puppet Master:

    puppet cert clean agent-server.com

  • Restart Puppet Master:

    /etc/init.d/puppetmaster restart

  • Run puppet agent:

    puppet agent -t

Error messages:

Agent:

root@agent-server:~# puppet agent -t
info: Creating a new SSL key for agent-server.com
info: Caching certificate for ca
info: Creating a new SSL certificate request for agent-server.com
info: Certificate Request fingerprint (md5): F2:2A:AD:3C:D5:E8:13:82:1D:C5:80:B4:FD:23:C4:86
info: Caching certificate for agent-server.com
info: Caching certificate_revocation_list for ca
err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
err: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client 

Puppet Master

[2017-08-29 18:08:51] 10.88.0.208 - - [29/Aug/2017:18:08:51 UTC] "GET /production/certificate/ca? HTTP/1.1" 200 1939
[2017-08-29 18:08:51] - -> /production/certificate/ca?
[2017-08-29 18:08:51] 10.88.0.208 - - [29/Aug/2017:18:08:51 UTC] "GET /production/certificate/agent-server.com? HTTP/1.1" 404 78
[2017-08-29 18:08:51] - -> /production/certificate/agent-server.com?
[2017-08-29 18:08:51] 10.88.0.208 - - [29/Aug/2017:18:08:51 UTC] "GET /production/certificate_request/agent-server.com? HTTP/1.1" 404 86
[2017-08-29 18:08:51] - -> /production/certificate_request/agent-server.com?
[2017-08-29 18:08:51] 10.88.0.208 - - [29/Aug/2017:18:08:51 UTC] "PUT /production/certificate_request/agent-server.com HTTP/1.1" 200 1448
[2017-08-29 18:08:51] - -> /production/certificate_request/agent-server.com
[2017-08-29 18:08:51] 10.88.0.208 - - [29/Aug/2017:18:08:51 UTC] "GET /production/certificate/agent-server.com? HTTP/1.1" 200 1448
[2017-08-29 18:08:51] - -> /production/certificate/agent-server.com?
[2017-08-29 18:08:56] 10.88.0.208 - - [29/Aug/2017:18:08:56 UTC] "GET /production/certificate_revocation_list/ca? HTTP/1.1" 200 11220
[2017-08-29 18:08:56] - -> /production/certificate_revocation_list/ca?
[2017-08-29 18:08:56] ERROR OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=SSLv3 read client certificate A: sslv3 alert certificate revoked
/usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:35:in `accept'
/usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:35:in `block (2 levels) in listen'
/usr/lib/ruby/1.9.1/webrick/server.rb:191:in `call'
/usr/lib/ruby/1.9.1/webrick/server.rb:191:in `block in start_thread'
[2017-08-29 18:08:56] ERROR OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=SSLv3 read client certificate A: sslv3 alert certificate revoked
/usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:35:in `accept'
/usr/lib/ruby/vendor_ruby/puppet/network/http/webrick.rb:35:in `block (2 levels) in listen'
/usr/lib/ruby/1.9.1/webrick/server.rb:191:in `call'
/usr/lib/ruby/1.9.1/webrick/server.rb:191:in `block in start_thread

So judging by the logs it looks like the certificate is being revoked, even though it is a brand new one. In addition, it can't be a time issue because the two servers are very close, only apart by 2-3 seconds.

I'm rather stumped, unfortunately. Any help is appreciated.

0

The server is refusing your client certificate as it was sign by old server CA. Remove or move your old client certificate from the client host

sudo mv /var/lib/puppet/ssl /var/lib/puppet/ssl.old

then trigger again the agent from the client host.

Also server log indicates it could come from time mismatch between host. ensure both host are having the same time and are sync with a NTP server.

| improve this answer | |
  • Hi, if you read my post you'd see that I already tried that, and that it's not a time mismatch issue – fish_king Sep 12 '17 at 9:21
  • Did you tried to open, with external tools, the certificate from the agent and the one from the server to check their validaties, CA etc... ? – DevOps Sep 12 '17 at 12:57
  • Yes. They are valid. – fish_king Sep 13 '17 at 8:53
  • I tried to reproduce the issue with other versions (3.8.7) but everything was fine. However, it made me realise that you use different versions between master and agent. This is not recommended and could be the cause of the issue. – DevOps Sep 14 '17 at 9:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.