Accidentally revoked all puppetmaster certs

Question

I ran puppet cert clean --all, thinking it would only clear the certificates that hadn't been signed yet. There's about 300 nodes that rely on the puppet master. As far as I can tell, puppet agent still works on them, but I think that's because there's a cached copy of the certificate somewhere.

Any way to rectify the situation without having to manually log into 300 different servers?

Thanks

EDIT: I should mention that /var/lib/puppet wasn't being backed up for some reason.

Answer 1

If they're still checking in, that's probably because the puppet master is not checking the CRL; they might not exist in the cert inventory on the master any more but they're still signed by the CA. Revoking trumps that but the revoke doesn't seem to be preventing their agent runs (verify they aren't just using cached catalogs with puppet agent --test).

Because of this, you should be able to do some creative configuration management to get them to enroll for new certificates - say, maybe something like this..

exec { 'ssl hackery':
  command  => '/bin/mv /var/lib/puppet/ssl /var/lib/puppet/ssl_old',
  creates  => '/var/lib/puppet/ssl_old',
}

(Test this thoroughly on a single host before hitting all the nodes with it, or you will indeed be touching each one!)

Answer 2

Log into 300 different servers in a loop?

for host in `cat allmypuppetboxes`; do
  ssh -o ConnectTimeout=5 root@$host '/etc/init.d/puppet stop ; rm -rf /var/lib/puppet/ssl /var/lib/puppet/ssl.expired ; puppet agent --server puppet.example.com --test --waitforcert 5'
done