PuppetDB: Failed to submit 'replace facts' command

Question

I recently revoked/cleaned a Puppet agent cert, and this seems to have negative effects in PuppetDB. I see a bug has been filed here with some instructions on fixing the issue. A user had a similar issue here, but none of this is working for me.

The server is running CentOS 6.2, Puppet 2.7.13, and Puppet DB 0.9. The error is:

root@harp:/etc/puppetdb/ssl> puppet agent --test
err: Cached facts for harp failed: Failed to find facts from PuppetDB at harp.mydomain.com:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client
info: Loading facts in /etc/puppet/modules/dns/lib/facter/datacenter.rb
info: Caching facts for harp
err: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client
err: Could not run Puppet configuration client: Could not retrieve local facts: Failed to submit 'replace facts' command for harp to PuppetDB at harp.mydomain.com:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client

NTP is working properly from what I see and the datetime looks good. "harp" is actually the puppet master server, so there shouldn't be an issue with time between the agent and server here since they're the same.

Old certificate:

root@harp:/etc/puppetdb/ssl> puppet cert list --all
+ harp  (DF:8F:65:36:58:4C:DE:66:2B:65:D1:E6:18:B7:F2:33)

Clean and generate new cert for agent:

root@harp:/etc/puppetdb/ssl> puppet cert clean harp
notice: Revoked certificate with serial 18
notice: Removing file Puppet::SSL::Certificate harp at '/var/lib/puppet/ssl/ca/signed/harp.pem'
notice: Removing file Puppet::SSL::Certificate harp at '/var/lib/puppet/ssl/certs/harp.pem'
notice: Removing file Puppet::SSL::CertificateRequest harp at '/var/lib/puppet/ssl/certificate_requests/harp.pem'
notice: Removing file Puppet::SSL::Key harp at '/var/lib/puppet/ssl/private_keys/harp.pem'

root@harp:/etc/puppetdb/ssl> puppet agent --test
info: Creating a new SSL key for harp
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
info: Creating a new SSL certificate request for harp
info: Certificate Request fingerprint (md5): 72:5E:99:6A:DE:B0:76:BD:1A:7D:FD:DC:A9:E8:71:AD
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
Exiting; no certificate found and waitforcert is disabled

root@harp:/etc/puppetdb/ssl> puppet cert list
  harp (72:5E:99:6A:DE:B0:76:BD:1A:7D:FD:DC:A9:E8:71:AD)

root@harp:/etc/puppetdb/ssl> puppet cert sign harp
notice: Signed certificate request for harp
notice: Removing file Puppet::SSL::CertificateRequest harp at '/var/lib/puppet/ssl/ca/requests/harp.pem'

root@harp:/etc/puppetdb/ssl> puppet cert list --all
+ harp  (4A:D4:90:87:15:1B:D3:FD:A8:15:D9:C0:FB:08:5C:79)

root@harp:/etc/puppetdb/ssl> service puppetdb restart
Stopping puppetdb: /etc/init.d/puppetdb: line 77: kill: (8623) - No such process
                                                           [FAILED]
Starting puppetdb:                                         [  OK  ]

OK then, restart again for good measure:

root@harp:/etc/puppetdb/ssl> service puppetdb restart
Stopping puppetdb:                                         [  OK  ]
Starting puppetdb:                                         [  OK  ]

Run the SSL configuration script

root@harp:/etc/puppetdb/ssl> /usr/sbin/puppetdb-ssl-setup
cp: cannot stat `/var/lib/puppet/ssl/certs/harp.pem': No such file or directory

root@harp:/etc/puppetdb/ssl> ls -la /var/lib/puppet/ssl/certs
total 12
drwxr-xr-x 2 puppet root 4096 Jun 19 07:19 ./
drwxrwx--x 8 puppet root 4096 Apr 24 10:04 ../
-rw-r--r-- 1 puppet root 1854 Apr 24 10:04 ca.pem

OK then, try again for good measure:

root@harp:/etc/puppetdb/ssl> /usr/sbin/puppetdb-ssl-setup
Certificate was added to keystore
Usage: pkcs12 [options]
where options are
-export       output PKCS12 file
-chain        add certificate chain
-inkey file   private key if not infile
-certfile f   add all certs in f
-CApath arg   - PEM format directory of CA's
-CAfile arg   - PEM format file of CA's
-name "name"  use name as friendly name
-caname "nm"  use nm as CA friendly name (can be used more than once).
-in  infile   input filename
...snip...
-CSP name     Microsoft CSP name
-LMK          Add local machine keyset attribute to private key

It does not appear that the keystores in /etc/puppetdb/ssl have changed/regenerated. At this point, running puppet agent --test results in the same errors, and restarting puppet and puppetdb do not help.

Keystore info:

root@harp:/etc/puppetdb/ssl> keytool -list -keystore /etc/puppetdb/ssl/keystore.jks
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

harp.mydomain.com, May 25, 2012, PrivateKeyEntry,
Certificate fingerprint (MD5): 06:A8:D3:2A:70:F3:6D:34:62:91:45:22:8A:C4:A8:86
root@harp:/etc/puppetdb/ssl> keytool -list -keystore /etc/puppetdb/ssl/truststore.jks
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

puppetdb ca, May 25, 2012, trustedCertEntry,
Certificate fingerprint (MD5): 13:AD:D8:BC:42:40:47:BB:D2:5C:ED:3C:D1:78:26:88
root@harp:/etc/puppetdb/ssl> puppet cert --fingerprint ca harp.mydomain.com
ca 13:AD:D8:BC:42:40:47:BB:D2:5C:ED:3C:D1:78:26:88
err: Could not call fingerprint: Could not find a certificate or csr for harp.mydomain.com

root@harp:/etc/puppetdb/ssl> puppet cert --fingerprint ca harp
ca 13:AD:D8:BC:42:40:47:BB:D2:5C:ED:3C:D1:78:26:88
harp 4A:D4:90:87:15:1B:D3:FD:A8:15:D9:C0:FB:08:5C:79

How can I get the puppetdb keystore to actually regenerate? I tried deleting the files in /etc/puppetdb/ssl/, but no luck.

Answer 1

I got it going, but can't say exactly what steps were necessary or not.

This issue started because authentication on several hosts was slow or hanging, and appeared to be related to domain controller/DNS cache issues. Removing domain mydomain.com entry from /etc/resolv.conf on the puppet master and agents solved the issue, but that created issues with existing puppet certs. I ran puppet cert clean --all on the master to try and recreate all certs, but this did not play well with PuppetDB.

Solution

Clean out old certs on master:

puppet cert clean --all

Clean out old certs on all agents:

rm -rf /var/lib/puppet/ssl

Recreate PuppetDB keystores:

facter fqdn is not available after removing domain foo.com from /etc/resolv.conf. This causes puppetdb-ssl-setup to fail silently.

Edit /usr/sbin/puppetdb-ssl-setup, add a piece of code to use just facter hostname if facter fqdn is empty:

# near line 10
fqdn=`facter fqdn`
# add this "if" section
if [ ! -n "$fqdn" ] ; then
  fqdn=`facter hostname`
fi

Permissions fix:

chown -R puppetdb:puppetdb /etc/puppetdb/ssl

Update passwords in /etc/puppetdb/conf.d/jetty.ini with new keystore/truststore passcode (same pass), which you can get from:

cat /etc/puppetdb/ssl/puppetdb_keystore_pw.txt

Restart puppetdb

service puppetdb restart

Then go to each agent and request new certs and sign each on the master.

Answer 2

This also happens, when your memory settings for puppetdb are too low.

vim /etc/default/puppetdb

Edit the line

JAVA_ARGS="-Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -Djava.security.egd=file:/dev/urandom"

should become

JAVA_ARGS="-Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -Djava.security.egd=file:/dev/urandom"

and restart puppetdb

sudo service puppetdb restart

Answer 3

Had a similar issue. Solution:

1.) remove the pe-puppetdb pid file on master 2.) stop the pe-puppetdb service on master 3.) start the pe-puppetdb service on master wait 30 seconds.

Answer 4

I had a similar issue after upgrading the puppet master (including puppetdb from 1.6.3 to 2.3.8) from 3.7.x to 3.8.x and got the following error message:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for puppet-client to PuppetDB at puppetmaster:8081: Connection refused - connect(2)

The solution for this was on the one hand to restart the puppetdb and on the other to also restart the puppet agent client. After that the agent was able to continue its work.