How can the little guys effectively learn and use Puppet?

Question

Six months ago, in our not-for-profit project we decided to start migrating our system management to a Puppet-controlled environment because we are expecting our number of servers to grow substantially between now and a year from now.

Since the decision has been made our IT guys have become a bit too annoyed a bit too often. Their biggest objections are:

• "We're not programmers, we're sysadmins";
• Modules are available online but many differ from one another; wheels are being reinvented too often, how do you decide which one fits the bill;
• Code in our repo is not transparent enough, to find how something works they have to recurse through manifests and modules they might have even written themselves a while ago;
• One new daemon requires writing a new module, conventions have to be similar to other modules, a difficult process;
• "Let's just run it and see how it works"
• Tons of hardly known 'extensions' in community modules: 'trocla', 'augeas', 'hiera'... how can our sysadmins keep track?

I can see why a large organisation would dispatch their sysadmins to Puppet courses to become Puppet masters. But how would smaller players get to learn Puppet to a professional level if they do not go to courses and basically learn it via their browser and editor?

Answer 1

I started using Puppet ahead of deploying a new infrastructure and simply bought a (well-regarded) book on the topic. I don't think most people actually obtain professional Puppet training. I worked on examples until I could mold the process to my environment. This was December 2011, so within a few weeks, I was able to understand the basics and get a production framework in place. I wasn't new to configuration management, having a CFEngine background, but many of your sysadmins' concerns resonate. I made mistakes and had to refactor several times, but I did get things working satisfactorily.

A few notes on your points...

• The traditional systems administration role is changing. Adapt or be left behind. I've been a successful systems engineer, but am having to retool as well (learning Python, for example). The focus on individual servers is diminished as hardware abstraction through virtualization and public and private cloud services gain traction. This means automation of systems tasks and the use of configuration management to wrest control of a larger number of servers. Add DevOps concepts to the mix, and you'll see that the customer/end-user expectations and requirements are changing.
• Puppet modules available online differ in style and structure and yes, I saw lots of overlap, redundancy and duplicated efforts. One developer I worked with said, "you could have developed your own tools in the time you spent looking online for something that works!" That gave me pause as I realized that Puppet seems to appeal more to developer types than admins looking for a best-practices or the right way approach.
• Document heavily in order to get a feel for how things are connected. Given the shaky definitions and lack of a standard way of doing things, your configuration management structure really is unique to your environment. That transparency is going to have to be developed within.
• I'd argue that it's reasonably easy to duplicate a module to accommodate a new daemon or add a service to an existing manifest, depending on how you've organized your servers and roles.
• I spent a lot of time testing on a single target before pushing changes to larger groups of servers. Running puppetd by hand on a representative server allowed me to debug changes and assess their impact. Maybe that's a bit conservative, but it was necessary.
• I'm not sure how much I'd depend on community modules. I did have to start using Augeas for some work, and lamented the fact that it was a functionality I took for granted in CFEngine.

In all, I feel like there isn't a well-defined standard when it comes to Puppet. I had trouble figuring out how to organize directory structure on my Puppetmaster, understanding how to manage certificate signing, getting proper reverse DNS in place everywhere, getting Puppet to scale appropriately for the environment and understanding when to leverage community modules versus building my own. It's a shift in thinking and I see how that would make a sysadmin panic. However, this was also solution built from scratch, so I had the luxury of evaluating tools. The decision to go this way was based on mindshare and the momentum behind Puppet. It was worth the effort to learn something new.

Remember, this site is a good resource, too.

Answer 2

On a previous job, I was assigned the task of doing a pilot implementation of Puppet. Now, I have programming background, though not Ruby, so I don't have as much problem as others do.

However, it's interesting to note that programmers without experience with non-traditional paradigms have problem with Puppet too, because Puppet is declarative, not imperative. In this sense, Puppet works pretty much like any configuration file: you say how things should be, and Puppet takes care of the rest.

After the pilot I had the opportunity to train a a dozen other admins with Puppet, plus giving presentations about it in two events. My take from that experience is that some admins took to it, and some didn't. These were all traditional admins, without programming skills, and varying levels of expertise.

One particular thing I noticed is that Puppet requires constant practice. People who were trained, wrote modules, and then spent a whole month or two doing something else came back to Puppet with little useful skill. People who kept doing small things in it every week never lost the ability.

Based on these two observations, I recommend you make sure everyone keeps adding some puppet class, definition or module every week (preferably at least two or three times). Those who still can't get used to it might really lack the skill to do it.

Then again, if Puppet was imposed on them from higher up, they might simply be reacting to what they perceive as a management intruding in how they do their work -- which would be true enough, in fact. It might be the case that letting them choose which configuration management system to use would improve things. Here are some alternatives:

• ANSIBLE: This is new, but it's based on shell commands and ssh, which might entice it to traditional sysadmins.
• Chef: Maybe their problem is declarative style, in which case Chef would be better if they have Ruby experience.
• Cuisine: A mix between ANSIBLE and Chef, but based on Python.
• CFEngine: old, fast, traditional -- it might win them over on the those grounds.

Answer 3

I've been using Puppet a bit over two years in small shops where I was the only sysadmin. The biggest hurdle I've had is learning how to develop software properly. There wasn't a week that had gone by where I didn't screw up something that I've told developers not to do a dozen times. I checked too much code in, I didn't break up checkins, I didn't tag, I didn't branch, didn't run the syntax checker, didn't use a standard, etc. If you're just starting out I'd recommend some of the following.

1. Realize you're developing software which you either don't know how to do or are doing poorly. This is expected because it's new.
2. Infrastructure as code is the reality and once you get over the hump it's quite powerful. I'd invite some developers over, show them your current development process (or lack thereof), don't take offense when they raise eyebrows, and take their suggests seriously. I'd recommend using whatever system and process your developers use unless it's completely inappropriate.
3. Puppet third party modules suck 90% of the time. I'd read them. I'd steal ideas from them. I would not pull them into my system without major edits. However I would pull in the puppet stdlib which adds some nice functionality.
4. augeas and hiera. Learn those two. The first allows complex editing of existing files in place. The second is an external data store.
5. Separate code from data. This is one of the harder concepts to learn. Hardcoding values like Monitoring Hosts into your module code is bad. Putting them in a data store (db, yaml (Hiera uses this be default), csv, whatever) that your modules can consume is good. An example is a webapp that uses Mysql. What this allows is the ability to push code and data separately. This makes your development process simpler.
6. puppet parser validate and puppet-lint as part of you pre or post code checkin process. Also rspec tests might be a good idea once you're up to speed.
7. write a style guide/code standard and use it. "where is the code that installs Apache" is a common problem. If your modules are mostly the same it should be easy.

In summary I've hit all of these problems and so have most of my sysadmin friends. It'll take some time to get good at using a config management system. Once you do you'll wonder how you ever lived without one. "Log into a server and make changes manually? Ick."

Answer 4

Six months ago, in our not-for-profit project we decided to start migrating our system management to a Puppet-controlled environment because we are expecting our number of servers to grow substantially between now and a year from now.

Sounds like a darn good idea to start early - Puppet is more than just config management, it's a form of documentation.

Since the decision has been made our IT guys have become a bit too annoyed a bit too often.

They need an attitude adjustment.

"We're not programmers, we're sysadmins";

Again, attitude. You -can- make a conf file for a server right? You can ease into the templating/'programmer' stuff as your needs and complexity evolves.

Modules are available online but many differ from one another; wheels are being reinvented too often, how do you decide which one fits the bill;

Tough one to answer - I always prefer the puppetlabs modules over most - and even at that, I don't use that many. Judgement call for sure. In my opinion, some modules are 'too frilly'.

Code in our repo is not transparent enough, to find how something works they have to recurse through manifests and modules they might have even written themselves a while ago;

This doesn't sound like a puppet problem, but more so an organizational or documentation issue?

One new daemon requires writing a new module, conventions have to be similar to other modules, a difficult process;

That daemon could be a class if it's simple enough to manage. I'm not sure what you mean by conventions, puppet enforces conventions on you pretty well doesn't it? Or are we talking along the lines of code formatting?

"Let's just run it and see how it works"

Not that bad of an idea if you take it slow and safe. I'd still start with a VM to get the gist of things.

Tons of hardly known 'extensions' in community modules: 'trocla', 'augeas', 'hiera'... how can our sysadmins keep track?

postfix, exim, sendmail, mysql, postgresql, iftop, iptraf, perl, perl modules.. Pick what you want and use it? I guess this sound more like an attitude thing again...

I can see why a large organisation would dispatch their sysadmins to Puppet courses to become Puppet masters. But how would smaller players get to learn Puppet to a professional level if they do not go to courses and basically learn it via their browser and editor?

I haven't attended any courses - while I am a programmer more than a sysadmin, I found it didn't need much programming skill to get anything accomplished.

The Puppet documentation, when followed is quite thorough. Just pay attention to the built-in types and spend some time looking at how other modules are put together. I wouldn't say it's -super- easy, but it's not -hard- either. It is a bit time consuming to get your infrastructure ready for puppet, but the time invested is assured to be well spent when you expand.

Answer 5

KISS (Keep it simple stupid) - Don't use new technologies just because they are there rather because you have a requirement for them, use the bare minimum that your deployment requires, update as required don't try to keep up with the bleeding edge. If you start with a basic setup and build on that it is easier to pickup as you go, and they shouldn't need a course (are these even available?).

The other area you can look at are your sysadmins. If they can't program as well, then are they advanced enough for a large deployment, where most of the work needs to be scripted whatever tools you use?

Answer 6

I work for a non profit as well and was responsible for initially bringing Linux boxes in house and shortly thereafter Puppet for managing them. There are a couple specific things we've done which have really helped get things rolling.

First and foremost I've tried to stay away from the third party modules. The inbuilt tools handle 90% of our management. The biggest third party utility I use is the firewall module. Any custom facts, etc are developed with the whole team involved. We developed a template module and keep the file management, package, services, etc all standardized off this template.

Second, after standardizing on using the inbuilt modules we started using Git and Atlassian's Crucible - free for non profits, by the way - for performing reviews of all configuration changes. This provides the transparency desired.

Third, I automated the setup for Puppet so that new hosts can be added automatically with a default set of options. There are several ways of addressing this. Since I already had a complete Kickstart environment I opted to add a script there.