I am currently trying to setup a load balanced puppet infrastructure. I am using puppetserver for the puppet server(s) and puppet agent 3.7.3 for the clients.

Currently I have four servers setup in a local DNS server and name resolution is working correctly (I have not shown the FDQN's for brevity)

vmhgmaasdns01  IN A
vmhgmaasmgmt01  IN A
vmhgmaasproxy01  IN A
vmhgmaaspuppetdb01  IN A
mgmt  IN CNAME vmhgmaasproxy01

All of the servers are able to connect to the puppet server vmhgmaasmgmt01 when doing a puppet run using --server=vmhgmaasmgmt01

However when trying to use --server=mgmt I am getting errors

puppet agent --no-daemonize --no-splay --verbose --onetime --server=mgmt
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: end of file reached
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': Broken pipe - SSL_connect
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://mgmt/pluginfacts: Broken pipe - SSL_connect
Wrapped exception:
Broken pipe - SSL_connect
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': Broken pipe - SSL_connect
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://mgmt/plugins: Broken pipe - SSL_connect
Wrapped exception:
Broken pipe - SSL_connect
Info: Loading facts
Error: Could not retrieve catalog from remote server: Broken pipe 

I can run

openssl s_client -connect mgmt:8140 -cert  /etc/puppet/ssl/certs/vmhgmaasproxy01.pem -key ssl/private_keys/vmhgmaasproxy01.pem -CAfile ssl/certs/ca.pem

Which shows I can successfully verify the SSL connection through the load balancer

New, TLSv1/SSLv3, Cipher is AES256-SHA256
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
    Protocol  : TLSv1.2
    Cipher    : AES256-SHA256
    Session-ID: 54C623F7CC73B6CEDFC8C6BF1366FE96049030E60667FE170113D30EA2221F06
    Master-Key: B19715E32AE17A2C7D501D80A9D695C476A99CFB5441D07142650689CD554418C193505A5468364A7E0F482304F32C1E
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1422271478
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)

I have setup /etc/puppet/puppet.conf options to point to the certificates and keys used in the above example.

In HAProxy I have the front-end and back-ends set in TCP mode and when I try to run the puppet run I can see the sessions increasing in the front-end and back-end in the stats page. I can also see that the requests are being received by the puppet server using tcpdump.

The puppetserver certificate have been generated with dns_alt_names for the mgmt loadbalancer as well as the dns name for the host.

I can see nothing in the log files in /var/log/puppetserver/puppetserver.log with log level set to DEBUG for the failed connections.

All servers are running CentOS 6.6 and I have turned SELinux into permissive mode.

Any help greatly appreciated as I have tried and failed over the last three days to follow details from the sparse official documentation

  • Did you solve this issue? – 030 Feb 13 '15 at 18:15

I was able to resolve the issue by setting up the proxy server to use SSL termination to enable layer 7 load balancing and http mode.

# This file managed by Puppet
  group  haproxy
  log  192.168.*.* local1
  maxconn  4000
  nbproc  1
  pidfile  /var/run/haproxy.pid
  user  haproxy

  maxconn  8000
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s

frontend puppet-frontend
  bind 192.168.*.*:8140 ssl crt /etc/haproxy/ssl/cert.pem ca-file /etc/haproxy/ssl/ca.pem verify none
  mode  http
  acl ca path -m sub certificate  
  default_backend puppet-backend  
  use_backend puppet-ca-backend if ca  

backend puppet-backend
  balance roundrobin  
  mode http  
  stick on src  
  stick-table type ip size 1m expire 1m  
  server vmhgmaasmgmt01 192.168.*.*:8140 check
  server vmhgmaasmgmt02 192.168.*.*:8140 check

backend puppet-ca-backend
  mode http  
  server vmhgmaasmgmt01 192.168.*.*:8140 check

This allows the catalogs to goto either of the two puppet servers but certificate requests are routed to the CA. I had to modify webserver.conf to use port and host instead of ssl-host and ssl-port and add the following to master.conf

master: {
    allow-header-cert-info: true

biggest PITA was not realizing that I had to combine the key and certs for HAproxy into a PEM file. (not in all of the above I have added . to obfuscate the real ips)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.