Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Six months ago, in our not-for-profit project we decided to start migrating our system management to a Puppet-controlled environment because we are expecting our number of servers to grow substantially between now and a year from now.

Since the decision has been made our IT guys have become a bit too annoyed a bit too often. Their biggest objections are:

  • "We're not programmers, we're sysadmins";
  • Modules are available online but many differ from one another; wheels are being reinvented too often, how do you decide which one fits the bill;
  • Code in our repo is not transparent enough, to find how something works they have to recurse through manifests and modules they might have even written themselves a while ago;
  • One new daemon requires writing a new module, conventions have to be similar to other modules, a difficult process;
  • "Let's just run it and see how it works"
  • Tons of hardly known 'extensions' in community modules: 'trocla', 'augeas', 'hiera'... how can our sysadmins keep track?

I can see why a large organisation would dispatch their sysadmins to Puppet courses to become Puppet masters. But how would smaller players get to learn Puppet to a professional level if they do not go to courses and basically learn it via their browser and editor?

share|improve this question

6 Answers 6

up vote 74 down vote accepted

I started using Puppet ahead of deploying a new infrastructure and simply bought a (well-regarded) book on the topic. I don't think most people actually obtain professional Puppet training. I worked on examples until I could mold the process to my environment. This was December 2011, so within a few weeks, I was able to understand the basics and get a production framework in place. I wasn't new to configuration management, having a CFEngine background, but many of your sysadmins' concerns resonate. I made mistakes and had to refactor several times, but I did get things working satisfactorily.

A few notes on your points...

  • The traditional systems administration role is changing. Adapt or be left behind. I've been a successful systems engineer, but am having to retool as well (learning Python, for example). The focus on individual servers is diminished as hardware abstraction through virtualization and public and private cloud services gain traction. This means automation of systems tasks and the use of configuration management to wrest control of a larger number of servers. Add DevOps concepts to the mix, and you'll see that the customer/end-user expectations and requirements are changing.
  • Puppet modules available online differ in style and structure and yes, I saw lots of overlap, redundancy and duplicated efforts. One developer I worked with said, "you could have developed your own tools in the time you spent looking online for something that works!" That gave me pause as I realized that Puppet seems to appeal more to developer types than admins looking for a best-practices or the right way approach.
  • Document heavily in order to get a feel for how things are connected. Given the shaky definitions and lack of a standard way of doing things, your configuration management structure really is unique to your environment. That transparency is going to have to be developed within.
  • I'd argue that it's reasonably easy to duplicate a module to accommodate a new daemon or add a service to an existing manifest, depending on how you've organized your servers and roles.
  • I spent a lot of time testing on a single target before pushing changes to larger groups of servers. Running puppetd by hand on a representative server allowed me to debug changes and assess their impact. Maybe that's a bit conservative, but it was necessary.
  • I'm not sure how much I'd depend on community modules. I did have to start using Augeas for some work, and lamented the fact that it was a functionality I took for granted in CFEngine.

In all, I feel like there isn't a well-defined standard when it comes to Puppet. I had trouble figuring out how to organize directory structure on my Puppetmaster, understanding how to manage certificate signing, getting proper reverse DNS in place everywhere, getting Puppet to scale appropriately for the environment and understanding when to leverage community modules versus building my own. It's a shift in thinking and I see how that would make a sysadmin panic. However, this was also solution built from scratch, so I had the luxury of evaluating tools. The decision to go this way was based on mindshare and the momentum behind Puppet. It was worth the effort to learn something new.

Remember, this site is a good resource, too.

share|improve this answer
14  
I went from no experience on Puppet to having my complete environment managed in two weeks flat. I'm responsible for ~40 virtual machines, albeit all running Ubuntu. That simplified things quite a bit. I'm a developer by profession. "Adapt or be left behind" - I'm now devops + sysadmin + architect. Excellent answer! –  François Beausoleil Jun 6 '12 at 13:58
    
I would recommend them to start deploying small services, first in standalone then start tinkering with more servers. I do not have to work with Puppet, but I have a small VPS and I recently made my own Puppet modules. If they want to keep up with the rest of sysadmins in the current century, they had better be open-minded. I do it because I like, and I guess not everybody likes learning new things, but one thing is for sure, nowadays sysadmins are closer to developers than ever. –  Sergio Galvan Jun 7 '12 at 19:22
1  
I work in a small company and I also run puppetd -t for testing on a couple boxes before pushing to all the servers. It never fails that a couple have something unique that causes my updates to fail on them. Puppet is a lot easier when you have a controlled and consistent environment for the beginning. –  jordanm Aug 25 '12 at 5:33
    
@ewwhite, I've worked my way through the Puppet tutorial in their docs but was wondering what book you used when learning? I kind of feel like the tutorial provided in the docs was missing something that is keeping everything from clicking with me as I am working with Puppet on test hosts to learn what I'm doing. EDIT: Or any additional resources you can recommend. Thanks. –  Mike Keller Nov 9 '12 at 2:32
1  
@MikeKeller I liked to it in my post... But it's available here. –  ewwhite Nov 9 '12 at 2:41

On a previous job, I was assigned the task of doing a pilot implementation of Puppet. Now, I have programming background, though not Ruby, so I don't have as much problem as others do.

However, it's interesting to note that programmers without experience with non-traditional paradigms have problem with Puppet too, because Puppet is declarative, not imperative. In this sense, Puppet works pretty much like any configuration file: you say how things should be, and Puppet takes care of the rest.

After the pilot I had the opportunity to train a a dozen other admins with Puppet, plus giving presentations about it in two events. My take from that experience is that some admins took to it, and some didn't. These were all traditional admins, without programming skills, and varying levels of expertise.

One particular thing I noticed is that Puppet requires constant practice. People who were trained, wrote modules, and then spent a whole month or two doing something else came back to Puppet with little useful skill. People who kept doing small things in it every week never lost the ability.

Based on these two observations, I recommend you make sure everyone keeps adding some puppet class, definition or module every week (preferably at least two or three times). Those who still can't get used to it might really lack the skill to do it.

Then again, if Puppet was imposed on them from higher up, they might simply be reacting to what they perceive as a management intruding in how they do their work -- which would be true enough, in fact. It might be the case that letting them choose which configuration management system to use would improve things. Here are some alternatives:

  • ANSIBLE: This is new, but it's based on shell commands and ssh, which might entice it to traditional sysadmins.
  • Chef: Maybe their problem is declarative style, in which case Chef would be better if they have Ruby experience.
  • Cuisine: A mix between ANSIBLE and Chef, but based on Python.
  • CFEngine: old, fast, traditional -- it might win them over on the those grounds.
share|improve this answer
7  
The nice thing about ANSIBLE is that it works across galactic distances with absolutely no delay in data transmission! –  Kalamane Jun 6 '12 at 20:10
    
Thank you for the ANSIBLE mention. I wasn't aware of it until now. –  ewwhite Jun 7 '12 at 0:58
    
@ewwhite You're welcome. I, myself, have only recently discovered it, but much about it has caught my attention. If we hadn't so much in Puppet already, I'd definitely try it out. –  Daniel C. Sobral Jun 8 '12 at 3:50
1  
Add Python-based Saltstack (saltstack.com/community.html) to that list. And scrap cuisine, as its development has ceased. –  Willem Oct 16 '13 at 9:34

I've been using Puppet a bit over two years in small shops where I was the only sysadmin. The biggest hurdle I've had is learning how to develop software properly. There wasn't a week that had gone by where I didn't screw up something that I've told developers not to do a dozen times. I checked too much code in, I didn't break up checkins, I didn't tag, I didn't branch, didn't run the syntax checker, didn't use a standard, etc. If you're just starting out I'd recommend some of the following.

  1. Realize you're developing software which you either don't know how to do or are doing poorly. This is expected because it's new.
  2. Infrastructure as code is the reality and once you get over the hump it's quite powerful. I'd invite some developers over, show them your current development process (or lack thereof), don't take offense when they raise eyebrows, and take their suggests seriously. I'd recommend using whatever system and process your developers use unless it's completely inappropriate.
  3. Puppet third party modules suck 90% of the time. I'd read them. I'd steal ideas from them. I would not pull them into my system without major edits. However I would pull in the puppet stdlib which adds some nice functionality.
  4. augeas and hiera. Learn those two. The first allows complex editing of existing files in place. The second is an external data store.
  5. Separate code from data. This is one of the harder concepts to learn. Hardcoding values like Monitoring Hosts into your module code is bad. Putting them in a data store (db, yaml (Hiera uses this be default), csv, whatever) that your modules can consume is good. An example is a webapp that uses Mysql. What this allows is the ability to push code and data separately. This makes your development process simpler.
  6. puppet parser validate and puppet-lint as part of you pre or post code checkin process. Also rspec tests might be a good idea once you're up to speed.
  7. write a style guide/code standard and use it. "where is the code that installs Apache" is a common problem. If your modules are mostly the same it should be easy.

In summary I've hit all of these problems and so have most of my sysadmin friends. It'll take some time to get good at using a config management system. Once you do you'll wonder how you ever lived without one. "Log into a server and make changes manually? Ick."

share|improve this answer
    
Thank you for your suggestions, especially augeas and hiera are two components we have started to implement and this made us much more aware, even confident of Puppet's capabilities. So thanks :-) –  drumfire Jul 18 '12 at 0:14

KISS (Keep it simple stupid) - Don't use new technologies just because they are there rather because you have a requirement for them, use the bare minimum that your deployment requires, update as required don't try to keep up with the bleeding edge. If you start with a basic setup and build on that it is easier to pickup as you go, and they shouldn't need a course (are these even available?).

The other area you can look at are your sysadmins. If they can't program as well, then are they advanced enough for a large deployment, where most of the work needs to be scripted whatever tools you use?

share|improve this answer
2  
... because we are expecting our number of servers to grow substantially between now and a year from now. Requirement? –  Jeff Ferland Jun 6 '12 at 14:02
1  
Really depends on how certain that expectation is and if what you put in place will still be suitable by the time the need actually arises. –  JamesRyan Jun 6 '12 at 16:05
    
+1 for "use the bare minimum that your deployment requires" - Alot of the puppet issues i've run into stem from trying to make puppet control everything on the system. –  Sirex Jul 16 '12 at 23:40

Six months ago, in our not-for-profit project we decided to start migrating our system management to a Puppet-controlled environment because we are expecting our number of servers to grow substantially between now and a year from now.

Sounds like a darn good idea to start early - Puppet is more than just config management, it's a form of documentation.

Since the decision has been made our IT guys have become a bit too annoyed a bit too often.

They need an attitude adjustment.

"We're not programmers, we're sysadmins";

Again, attitude. You -can- make a conf file for a server right? You can ease into the templating/'programmer' stuff as your needs and complexity evolves.

Modules are available online but many differ from one another; wheels are being reinvented too often, how do you decide which one fits the bill;

Tough one to answer - I always prefer the puppetlabs modules over most - and even at that, I don't use that many. Judgement call for sure. In my opinion, some modules are 'too frilly'.

Code in our repo is not transparent enough, to find how something works they have to recurse through manifests and modules they might have even written themselves a while ago;

This doesn't sound like a puppet problem, but more so an organizational or documentation issue?

One new daemon requires writing a new module, conventions have to be similar to other modules, a difficult process;

That daemon could be a class if it's simple enough to manage. I'm not sure what you mean by conventions, puppet enforces conventions on you pretty well doesn't it? Or are we talking along the lines of code formatting?

"Let's just run it and see how it works"

Not that bad of an idea if you take it slow and safe. I'd still start with a VM to get the gist of things.

Tons of hardly known 'extensions' in community modules: 'trocla', 'augeas', 'hiera'... how can our sysadmins keep track?

postfix, exim, sendmail, mysql, postgresql, iftop, iptraf, perl, perl modules.. Pick what you want and use it? I guess this sound more like an attitude thing again...

I can see why a large organisation would dispatch their sysadmins to Puppet courses to become Puppet masters. But how would smaller players get to learn Puppet to a professional level if they do not go to courses and basically learn it via their browser and editor?

I haven't attended any courses - while I am a programmer more than a sysadmin, I found it didn't need much programming skill to get anything accomplished.

The Puppet documentation, when followed is quite thorough. Just pay attention to the built-in types and spend some time looking at how other modules are put together. I wouldn't say it's -super- easy, but it's not -hard- either. It is a bit time consuming to get your infrastructure ready for puppet, but the time invested is assured to be well spent when you expand.

share|improve this answer
    
FYI this is coming from someone who -just- finished getting their infrastructure ready to roll. So I have a fresh experience and can't say it was time wasted. –  thinice Jun 6 '12 at 19:05
    
As a recent starter myself, I recognize myself entirely in your comment. –  Martijn Heemels Jul 30 '12 at 14:21
1  
In my case, a change in attitude was indeed necessary. Ops love automation and often script things, so it's mostly a matter of using different tools. It's a cool feeling to see your Puppet manifest configure an entire machine or a new service from scratch. The fact that an error can impact multiple machines at once requires getting used to more rigorous testing, which can be annoying but is obviously a good thing. Experiment with Vagrant, rspec-puppet, puppet-lint, Geppetto, Git branches and other free tools and you'll soon discover your favorite workflow. –  Martijn Heemels Jul 30 '12 at 14:34
1  
Working with Puppet also helped me learn Ruby, which has come to replace Bash as my default system tools language. –  Martijn Heemels Jul 30 '12 at 14:36

I work for a non profit as well and was responsible for initially bringing Linux boxes in house and shortly thereafter Puppet for managing them. There are a couple specific things we've done which have really helped get things rolling.

First and foremost I've tried to stay away from the third party modules. The inbuilt tools handle 90% of our management. The biggest third party utility I use is the firewall module. Any custom facts, etc are developed with the whole team involved. We developed a template module and keep the file management, package, services, etc all standardized off this template.

Second, after standardizing on using the inbuilt modules we started using Git and Atlassian's Crucible - free for non profits, by the way - for performing reviews of all configuration changes. This provides the transparency desired.

Third, I automated the setup for Puppet so that new hosts can be added automatically with a default set of options. There are several ways of addressing this. Since I already had a complete Kickstart environment I opted to add a script there.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.