I ran puppet cert clean --all, thinking it would only clear the certificates that hadn't been signed yet. There's about 300 nodes that rely on the puppet master. As far as I can tell, puppet agent still works on them, but I think that's because there's a cached copy of the certificate somewhere.

Any way to rectify the situation without having to manually log into 300 different servers?


EDIT: I should mention that /var/lib/puppet wasn't being backed up for some reason.

  • 3
    Restore the puppet certificate store from your backup. – Sven Jul 24 '14 at 20:35
  • 1
    I should mention that /var/lib/puppet wasn't being backed up for some reason. On the bright side, having to log into 300 servers manually should server as a helpful reminder about the importance of taking and testing backups. – HopelessN00b Jul 24 '14 at 20:41
  • As far as I can tell, puppet agent still works on them, If your agents are still working, after using clean --all that may mean you don't have your puppet master properly setup for certificate revocation. Issuing a clean is supposed to update the CRL. – Zoredache Jul 24 '14 at 20:56
  • Does puppet agent --test work? It's likely they're just getting the cached puppet config. – freiheit Jul 24 '14 at 21:11

If they're still checking in, that's probably because the puppet master is not checking the CRL; they might not exist in the cert inventory on the master any more but they're still signed by the CA. Revoking trumps that but the revoke doesn't seem to be preventing their agent runs (verify they aren't just using cached catalogs with puppet agent --test).

Because of this, you should be able to do some creative configuration management to get them to enroll for new certificates - say, maybe something like this..

exec { 'ssl hackery':
  command  => '/bin/mv /var/lib/puppet/ssl /var/lib/puppet/ssl_old',
  creates  => '/var/lib/puppet/ssl_old',

(Test this thoroughly on a single host before hitting all the nodes with it, or you will indeed be touching each one!)

  • If the master is checking the CRL, you could disable CRL checking temporarily. – Zoredache Jul 24 '14 at 21:10

Log into 300 different servers in a loop?

for host in `cat allmypuppetboxes`; do
  ssh -o ConnectTimeout=5 root@$host '/etc/init.d/puppet stop ; rm -rf /var/lib/puppet/ssl /var/lib/puppet/ssl.expired ; puppet agent --server puppet.example.com --test --waitforcert 5'
  • This is pretty much the second or third step of handling an expired puppetmaster cert... – freiheit Jul 24 '14 at 20:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.