0

I am currently trying to setup a load balanced puppet infrastructure. I am using puppetserver for the puppet server(s) and puppet agent 3.7.3 for the clients.

Currently I have four servers setup in a local DNS server and name resolution is working correctly (I have not shown the FDQN's for brevity)

vmhgmaasdns01  IN A 192.168.207.208
vmhgmaasmgmt01  IN A 192.168.207.210
vmhgmaasproxy01  IN A 192.168.207.209
vmhgmaaspuppetdb01  IN A 192.168.207.206
mgmt  IN CNAME vmhgmaasproxy01

All of the servers are able to connect to the puppet server vmhgmaasmgmt01 when doing a puppet run using --server=vmhgmaasmgmt01

However when trying to use --server=mgmt I am getting errors

puppet agent --no-daemonize --no-splay --verbose --onetime --server=mgmt
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: end of file reached
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': Broken pipe - SSL_connect
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://mgmt/pluginfacts: Broken pipe - SSL_connect
Wrapped exception:
Broken pipe - SSL_connect
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': Broken pipe - SSL_connect
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://mgmt/plugins: Broken pipe - SSL_connect
Wrapped exception:
Broken pipe - SSL_connect
Info: Loading facts
Error: Could not retrieve catalog from remote server: Broken pipe 

I can run

openssl s_client -connect mgmt:8140 -cert  /etc/puppet/ssl/certs/vmhgmaasproxy01.pem -key ssl/private_keys/vmhgmaasproxy01.pem -CAfile ssl/certs/ca.pem

Which shows I can successfully verify the SSL connection through the load balancer

New, TLSv1/SSLv3, Cipher is AES256-SHA256
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-SHA256
    Session-ID: 54C623F7CC73B6CEDFC8C6BF1366FE96049030E60667FE170113D30EA2221F06
    Session-ID-ctx:
    Master-Key: B19715E32AE17A2C7D501D80A9D695C476A99CFB5441D07142650689CD554418C193505A5468364A7E0F482304F32C1E
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1422271478
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)

I have setup /etc/puppet/puppet.conf options to point to the certificates and keys used in the above example.

In HAProxy I have the front-end and back-ends set in TCP mode and when I try to run the puppet run I can see the sessions increasing in the front-end and back-end in the stats page. I can also see that the requests are being received by the puppet server using tcpdump.

The puppetserver certificate have been generated with dns_alt_names for the mgmt loadbalancer as well as the dns name for the host.

I can see nothing in the log files in /var/log/puppetserver/puppetserver.log with log level set to DEBUG for the failed connections.

All servers are running CentOS 6.6 and I have turned SELinux into permissive mode.

Any help greatly appreciated as I have tried and failed over the last three days to follow details from the sparse official documentation

1
  • Did you solve this issue?
    – 030
    Feb 13, 2015 at 18:15

1 Answer 1

0

I was able to resolve the issue by setting up the proxy server to use SSL termination to enable layer 7 load balancing and http mode.

# This file managed by Puppet
global
  daemon  
  group  haproxy
  log  192.168.*.* local1
  maxconn  4000
  nbproc  1
  pidfile  /var/run/haproxy.pid
  user  haproxy

defaults
  maxconn  8000
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s

frontend puppet-frontend
  bind 192.168.*.*:8140 ssl crt /etc/haproxy/ssl/cert.pem ca-file /etc/haproxy/ssl/ca.pem verify none
  mode  http
  acl ca path -m sub certificate  
  default_backend puppet-backend  
  use_backend puppet-ca-backend if ca  

backend puppet-backend
  balance roundrobin  
  mode http  
  stick on src  
  stick-table type ip size 1m expire 1m  
  server vmhgmaasmgmt01 192.168.*.*:8140 check
  server vmhgmaasmgmt02 192.168.*.*:8140 check

backend puppet-ca-backend
  mode http  
  server vmhgmaasmgmt01 192.168.*.*:8140 check

This allows the catalogs to goto either of the two puppet servers but certificate requests are routed to the CA. I had to modify webserver.conf to use port and host instead of ssl-host and ssl-port and add the following to master.conf

master: {
    allow-header-cert-info: true
}

biggest PITA was not realizing that I had to combine the key and certs for HAproxy into a PEM file. (not in all of the above I have added . to obfuscate the real ips)

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .