1

I am using puppet agent v3.0.1 (On FreeBSD), trying to connect to a puppet master v3.1.1 (On Ubuntu) and the puppet agent hangs with the following debug output:

puppet agent --debug --no-daemonize
...
Debug: /File[/var/puppet/ssl/private_keys]: Autorequiring File[/var/puppet/ssl]
Debug: /File[/var/puppet/ssl/private]: Autorequiring File[/var/puppet/ssl]
Debug: /File[/var/puppet/ssl/private_keys/zg-3.XXXX.ch.pem]: Autorequiring         File[/var/puppet/ssl/private_keys]
Debug: /File[/var/puppet/ssl/public_keys/zg-3.XXXX.ch.pem]: Autorequiring File[/var/puppet/ssl/public_keys]
Debug: Finishing transaction 17214983260
[hangs.. about two minutes..]
Error: Could not request certificate: execution expired
Error: Could not request certificate: execution expired

The puppet master is hosted in a virtual machine and accessed via port forwarding over the VM host machine. I have tested access to the master from another node behind the firewall, which works correctly.

When the agent runs there is no indication on the puppet master that a connection has been made (although it's not clear that there should be).

  1. I have checked with netcat & telnet and confirmed that I able to initiate a connection from the agent to the server and exchange data.

  2. The same puppet master has worked correctly with a different FreeBSD agent running Puppet v3.0.2

  3. Running tcpdump during the agent run shows that some packets are going back and forth between the agent and server

  4. I have run truss on the (FreeBSD) puppet agent command and saw that the agent is getting the following error (twice):

    read(6,0x80711c3c6,1854) ERR#35 'Resource temporarily unavailable'

    File handle 6 is from a 'connect' system call to the correct IP address and port number (8140) of the master system.

I currently suspect the problem could be caused by one of the following:

  1. An agent/master puppet software compatibility issue

  2. SSL library compatibility problem

  3. Some strange firewall or NAT issue, which allows netcat/telnet communication, but not Puppet communication. A connection is being made as I get a different error when I run the agent but don't start the master

  4. A timing error or race condition

I don't think the problem is caused by:

  1. SSL certificate issue, as there ano certificate related errors. I've also tried deleting the contents of the ssl directory on both the agent and master systems

  2. Time synchronization problem, since both systems run NTP

Any hints on how to debug this?

Thanks in advance,

Alan Hodgkinson

P.S. Please ask for any addition (config) information necessary to help diagnose the problem.

  • Looking in the Puppet source code indicates that the agent hangs in puppet/ssl/host.rb in the certificate method. It appears that that agent it is unable to get a certificate, and that the certificate is returning nil, possibly because the variable ca (Certificate Authority) is nil. – Alan Hodgkinson May 10 '13 at 10:24
  • 1
    The problem was caused by small MTU size on the puppet master's ethernet port. The solution was to reduce the MTU size to 1400, with the command ip link set eth0 mtu 1400. The server was behind a firewall and communicated with outside world through a Cisco ADSL Router, which had an MTU size of 1492. We caught the problem by trying to scp a file of approx. 1500 bytes from the puppet server to the client. The scp created an empty file and then hung. This enabled us to eliminate puppet as the problem source. Strangely other servers in the same zone also have MTU zone = 1500 and work fine. – Alan Hodgkinson May 10 '13 at 16:23
  • Please write up your solution as an answer to this question and then accept that answer. That's a better way to mark a question as solved than just editing the subject line. – asciiphil May 13 '13 at 13:09
  • 1
    Welcome to Server Fault. Please do not tag questions with "SOLVED" or place answers in the question. Instead, post your own answer as an Answer below, and mark them solved by clicking the outline of the tick mark so that it turns green. – Michael Hampton Aug 2 '14 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.