Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a puppet master and slaves in different datacenters. The latency between them is ~40ms. When I run "puppet agent --test" on a slave to apply the latest manifest it takes ~360 seconds to finish.

After doing some digging I can see the main cause of the slow down is file transfers. It seems it's taking ~10 seconds to transfer each file. The files are only small (configuration files) so I can't understand why they would take so long.

This is an example of a file in my manifest:

file { "/etc/rsyncd.conf" :
    owner       => "root",
    group       => "root",
    mode        => 644,
    source  => "puppet:///files/rsyncd/rsyncd.conf"
}

Running puppet-profiler I see this:

10.21s - File[/etc/rsyncd.conf]

It also seems I cannot update more than one server at once using puppet. If I run two servers at the same time then puppet takes twice as long.

I have changed the puppet master from using webrick to mongrel, but this doesn't seem to help. This is making deploying changes painful. A simple config change can take an hour to roll out to all servers.

share|improve this question

4 Answers 4

Disclaimer: I am one of the Puppet developers.

When you source files with puppet:// URLs, Puppet will make two SSL connections back to the file server - one for the metadata, including checksum, and another for the content. The second is made, hopefully as expected, only if the content on disk is out of date.

The transfer itself is standard HTTPS, with nothing fancy going on, so beyond the SSL overhead and the latency cost of two connections, there is no reason that Puppet should be so slow.

Puppet will also checksum the file on both sides, the client and the server, when it comes to checking if it is up to date. That may be slow if your backing store is slow, too.

Finally, you should check if your network link is slow because it is saturated, or something like that. Wouldn't be the first time big buffers led to terrible network performance - and someone put a traffic shaper in the middle that made pings super-fast. ;)

Beyond that, I suggest you file a bug report. Including the details, and network traces, would be a big help in our being able to resolve this.

What you describe certainly isn't normal, and it isn't what we would expect to see for small configuration files on a reasonably fast network such as you describe. (Also, it isn't what others see, so clearly something unusual is happening in your setup.)

share|improve this answer
    
Thanks for your response, I'll fill out a bug report. Just for reference though, one server has GigE and the puppet master is connected at 100Mbit, so I don't think we're anywhere close to saturating the link. –  Noodles Jun 13 '12 at 3:04
    
@Noodles How about the response times of the puppet master to the HTTPS requests? Check its system load? Maybe its storage is slow? –  Shane Madden Jun 13 '12 at 3:22

In my experience older versions of puppet would serialize and DE-serialize all FILE data as they transferred it back and forth. And it was particularly bad and transferring large (many megabyte) binaries.

Mr Pittman suggests above that this may no longer be the case, but you may want to dig deeper and see if the version of puppet you're running has that optimization.

It looks like the change went in to puppet 0.25 - see bug 583 (http://projects.puppetlabs.com/issues/583) when they converted from xlrpc to rest.

My solution at the time was to use puppet to push rsync pre-shared keys and use an rsync daemon to serve up files (read-only) to end clients. Rsync can be quite efficient and transferring and updating lots of binary data.

So check your puppet versions, but also see if rsync can work for you as a performance comparison.

share|improve this answer
up vote 0 down vote accepted

I found the problem. It seems although I setup my Apache to run puppetmaster through passenger, the config file was never getting included in my httpd.conf. My puppetmaster was still running the default settings. (netstat -anp | grep 8140 showed that apache wasn't using this port).

After fixing these problems and making sure that Apache was listening on 8140 my puppet clients now take 8 seconds to run, down from 380 seconds.

share|improve this answer

Check the comment I left on http://projects.puppetlabs.com/issues/3365 about having more files on the target directory.

Saludos!

share|improve this answer
1  
Please sum up your comment and the content for the link. This url could fail to respond to requests for any number of reasons or the content could change in the future. –  tristan Apr 26 '13 at 19:29

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.