2

So, I am standing up a new server to replace an existing one. Should be easy, right? Revoke the old cert, create a new one and off you go. Here's the loop I am stuck in:

I've redacted the server names, cert fingerprint and domain. The servers shown below are:

  • Slave1 -- The machine that will be the partner of the one that is having issues. It is only mentioned below to prove one of the details.
  • Slave2 -- The machine that is giving me issues.
  • Master1 -- The puppet master (obviously)

On new build

[root@slave2 ~]# puppet agent -t
Error: Could not request certificate: The certificate retrieved from the master does not match the agent's private key.
Certificate fingerprint: XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:2F:F1
To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.
On the master:
puppet cert clean slave2.example.com
On the agent:
  rm -f /var/lib/puppet/ssl/certs/slave2.example.com.pem
  puppet agent -t

Exiting; failed to retrieve certificate and waitforcert is disabled

Okay, that's predictable and fully expected because this is a new server using an old name. Now on the master:

[root@master1 ~]# puppet cert clean slave2.example.com
Notice: Revoked certificate with serial 154

Note that there's nothing about the key files getting removed. This is because they are not there. Proof:

[root@master1 ~]# ls /var/lib/puppet/ssl/ca/signed/slave1.example.com.pem
/var/lib/puppet/ssl/ca/signed/slave1.example.com.pem
[root@master1 ~]# ls /var/lib/puppet/ssl/ca/signed/slave2.example.com.pem
ls: cannot access /var/lib/puppet/ssl/ca/signed/slave2.example.com.pem: No such file or directory

Okay, good. Now go back to the slave to complete the procedure by removing the .pem file and running puppet agent again:

[root@slave2 ~]# rm -f /var/lib/puppet/ssl/certs/slave2.example.com.pem
[root@slave2 ~]# puppet agent -t
Info: Caching certificate for slave2.example.com
Error: Could not request certificate: The certificate retrieved from the master does not match the agent's private key.
Certificate fingerprint: XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:2F:F1
To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.
On the master:
  puppet cert clean slave2.example.com
On the agent:
  rm -f /var/lib/puppet/ssl/certs/slave2.example.com.pem
  puppet agent -t

Exiting; failed to retrieve certificate and waitforcert is disabled

...and we are right back where we started with no change in outcome.

One last sanity check:

[root@master1 ~]# puppet cert list -a | grep -i slave2

...and there are no matches.

What am I doing wrong?

Addendum:

I'm inclined to believe that it is on the master, but not sure exactly how. Here's why:

[root@master1 ~]# puppet cert clean slave2.example.com
Notice: Revoked certificate with serial 154
[root@master1 ~]# puppet cert clean slave2.example.com
Notice: Revoked certificate with serial 154
[root@master1 ~]# puppet cert clean slave2.example.com
Notice: Revoked certificate with serial 154
[root@master1 ~]# puppet cert clean slave2.example.com
Notice: Revoked certificate with serial 154
[root@master1 ~]# puppet cert clean slave2.example.com
Notice: Revoked certificate with serial 154

Shouldn't that fail after the first time, because of the cert no longer being there?

  • try find /var/lib/puppet/ssl/ -type f -name '*slave2*' on master and on slave too – c4f4t0r Sep 12 at 1:00
  • @c4f4t0r Thanks for the suggestion. There were no results on master1. On slave2, I found the certs, but I have already (at someone else's suggestion) tried deleting that whole file tree (/var/lib/puppet/ssl) on slave2 to make it start over. It did not get me anwhere. – Glenn Lasher Sep 12 at 13:17
0

puppet ca is being deprecated though and may not work, but please try:

puppet ca destroy slave2.example.com

Also check the /var/lib/puppet/ssl/ca/requests directory for any pending request certificates manually even though puppet cert list -a isn't showing anything. Infact, do a ls -lRa /var/lib/puppet/ssl/ on the master and search for the slave2 cert, just to be sure.

  • Thank you for this suggestion. I did try this already yesterday, however, I ran it again just now to confirm. The response was "Nothing was deleted". – Glenn Lasher Sep 12 at 15:32
  • More things to try: # On Client: (Get the proper certname from client just to be sure using below cmd: ) puppet config print certname # On master: puppet node deactivate <certname> puppet node purge <certname> --- Also make sure the certname is set in puppet config as certname = slave2.example.com and then try running puppet: puppet agent -tv -- just some basics that is easy to miss out while cloning machines. :) – Aditya Pednekar 2 days ago
  • thank you for the suggestion. puppet config print certname does return the expected value. puppet node deactivate and puppet node purge are not valid on this version of Puppet (3.8.7). I did try puppet node clean slave2.example.com but it doesn't seem to have done anything useful. It leaves us still where we were. – Glenn Lasher 2 days ago
0

A Redditor provided a detailed step-by-step that seems to have resolved the issue. The three key things were:

  • Make sure that the clocks are in sync (they were)
  • Make sure that the puppet service is stopped (it wasn't)
  • Invoke puppet agent with an explicitly defined target for the master, e.g.
puppet agent -t --server master1.example.com

That combination of things got us past the cert issue.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.