I am currently trying to setup a load balanced puppet infrastructure. I am using puppetserver for the puppet server(s) and puppet agent 3.7.3 for the clients.
Currently I have four servers setup in a local DNS server and name resolution is working correctly (I have not shown the FDQN's for brevity)
vmhgmaasdns01 IN A 192.168.207.208 vmhgmaasmgmt01 IN A 192.168.207.210 vmhgmaasproxy01 IN A 192.168.207.209 vmhgmaaspuppetdb01 IN A 192.168.207.206 mgmt IN CNAME vmhgmaasproxy01
All of the servers are able to connect to the puppet server vmhgmaasmgmt01 when doing a puppet run using --server=vmhgmaasmgmt01
However when trying to use --server=mgmt I am getting errors
puppet agent --no-daemonize --no-splay --verbose --onetime --server=mgmt Warning: Unable to fetch my node definition, but the agent run will continue: Warning: end of file reached Info: Retrieving pluginfacts Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': Broken pipe - SSL_connect Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://mgmt/pluginfacts: Broken pipe - SSL_connect Wrapped exception: Broken pipe - SSL_connect Info: Retrieving plugin Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': Broken pipe - SSL_connect Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://mgmt/plugins: Broken pipe - SSL_connect Wrapped exception: Broken pipe - SSL_connect Info: Loading facts Error: Could not retrieve catalog from remote server: Broken pipe
I can run
openssl s_client -connect mgmt:8140 -cert /etc/puppet/ssl/certs/vmhgmaasproxy01.pem -key ssl/private_keys/vmhgmaasproxy01.pem -CAfile ssl/certs/ca.pem
Which shows I can successfully verify the SSL connection through the load balancer
New, TLSv1/SSLv3, Cipher is AES256-SHA256 Server public key is 4096 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.2 Cipher : AES256-SHA256 Session-ID: 54C623F7CC73B6CEDFC8C6BF1366FE96049030E60667FE170113D30EA2221F06 Session-ID-ctx: Master-Key: B19715E32AE17A2C7D501D80A9D695C476A99CFB5441D07142650689CD554418C193505A5468364A7E0F482304F32C1E Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1422271478 Timeout : 300 (sec) Verify return code: 0 (ok)
I have setup /etc/puppet/puppet.conf options to point to the certificates and keys used in the above example.
In HAProxy I have the front-end and back-ends set in TCP mode and when I try to run the puppet run I can see the sessions increasing in the front-end and back-end in the stats page. I can also see that the requests are being received by the puppet server using tcpdump.
The puppetserver certificate have been generated with dns_alt_names for the mgmt loadbalancer as well as the dns name for the host.
I can see nothing in the log files in /var/log/puppetserver/puppetserver.log with log level set to DEBUG for the failed connections.
All servers are running CentOS 6.6 and I have turned SELinux into permissive mode.
Any help greatly appreciated as I have tried and failed over the last three days to follow details from the sparse official documentation