110

Six months ago, in our not-for-profit project we decided to start migrating our system management to a Puppet-controlled environment because we are expecting our number of servers to grow substantially between now and a year from now.

Since the decision has been made our IT guys have become a bit too annoyed a bit too often. Their biggest objections are:

  • "We're not programmers, we're sysadmins";
  • Modules are available online but many differ from one another; wheels are being reinvented too often, how do you decide which one fits the bill;
  • Code in our repo is not transparent enough, to find how something works they have to recurse through manifests and modules they might have even written themselves a while ago;
  • One new daemon requires writing a new module, conventions have to be similar to other modules, a difficult process;
  • "Let's just run it and see how it works"
  • Tons of hardly known 'extensions' in community modules: 'trocla', 'augeas', 'hiera'... how can our sysadmins keep track?

I can see why a large organisation would dispatch their sysadmins to Puppet courses to become Puppet masters. But how would smaller players get to learn Puppet to a professional level if they do not go to courses and basically learn it via their browser and editor?

0

7 Answers 7

102

I started using Puppet ahead of deploying a new infrastructure and simply bought a (well-regarded) book on the topic. I don't think most people actually obtain professional Puppet training. I worked on examples until I could mold the process to my environment. This was December 2011, so within a few weeks, I was able to understand the basics and get a production framework in place. I wasn't new to configuration management, having a CFEngine background, but many of your sysadmins' concerns resonate. I made mistakes and had to refactor several times, but I did get things working satisfactorily.

A few notes on your points...

  • The traditional systems administration role is changing. Adapt or be left behind. I've been a successful systems engineer, but am having to retool as well (learning Python, for example). The focus on individual servers is diminished as hardware abstraction through virtualization and public and private cloud services gain traction. This means automation of systems tasks and the use of configuration management to wrest control of a larger number of servers. Add DevOps concepts to the mix, and you'll see that the customer/end-user expectations and requirements are changing.

  • Puppet modules available online differ in style and structure and yes, I saw lots of overlap, redundancy and duplicated efforts. One developer I worked with said, "you could have developed your own tools in the time you spent looking online for something that works!" That gave me pause as I realized that Puppet seems to appeal more to developer types than admins looking for a best-practices or the right way approach.

  • Document heavily in order to get a feel for how things are connected. Given the shaky definitions and lack of a standard way of doing things, your configuration management structure really is unique to your environment. That transparency is going to have to be developed within.

  • I'd argue that it's reasonably easy to duplicate a module to accommodate a new daemon or add a service to an existing manifest, depending on how you've organized your servers and roles.

  • I spent a lot of time testing on a single target before pushing changes to larger groups of servers. Running puppetd by hand on a representative server allowed me to debug changes and assess their impact. Maybe that's a bit conservative, but it was necessary.

  • I'm not sure how much I'd depend on community modules. I did have to start using Augeas for some work, and lamented the fact that it was a functionality I took for granted in CFEngine.

In all, I feel like there isn't a well-defined standard when it comes to Puppet. I had trouble figuring out how to organize directory structure on my Puppetmaster, understanding how to manage certificate signing, getting proper reverse DNS in place everywhere, getting Puppet to scale appropriately for the environment and understanding when to leverage community modules versus building my own. It's a shift in thinking and I see how that would make a sysadmin panic. However, this was also solution built from scratch, so I had the luxury of evaluating tools. The decision to go this way was based on mindshare and the momentum behind Puppet. It was worth the effort to learn something new.

Remember, this site is a good resource, too.

7
  • 20
    I went from no experience on Puppet to having my complete environment managed in two weeks flat. I'm responsible for ~40 virtual machines, albeit all running Ubuntu. That simplified things quite a bit. I'm a developer by profession. "Adapt or be left behind" - I'm now devops + sysadmin + architect. Excellent answer! Jun 6, 2012 at 13:58
  • I would recommend them to start deploying small services, first in standalone then start tinkering with more servers. I do not have to work with Puppet, but I have a small VPS and I recently made my own Puppet modules. If they want to keep up with the rest of sysadmins in the current century, they had better be open-minded. I do it because I like, and I guess not everybody likes learning new things, but one thing is for sure, nowadays sysadmins are closer to developers than ever. Jun 7, 2012 at 19:22
  • 2
    I work in a small company and I also run puppetd -t for testing on a couple boxes before pushing to all the servers. It never fails that a couple have something unique that causes my updates to fail on them. Puppet is a lot easier when you have a controlled and consistent environment for the beginning.
    – jordanm
    Aug 25, 2012 at 5:33
  • 1
    @ewwhite, I've worked my way through the Puppet tutorial in their docs but was wondering what book you used when learning? I kind of feel like the tutorial provided in the docs was missing something that is keeping everything from clicking with me as I am working with Puppet on test hosts to learn what I'm doing. EDIT: Or any additional resources you can recommend. Thanks. Nov 9, 2012 at 2:32
  • 1
    @MikeKeller I liked to it in my post... But it's available here.
    – ewwhite
    Nov 9, 2012 at 2:41
29

On a previous job, I was assigned the task of doing a pilot implementation of Puppet. Now, I have programming background, though not Ruby, so I don't have as much problem as others do.

However, it's interesting to note that programmers without experience with non-traditional paradigms have problem with Puppet too, because Puppet is declarative, not imperative. In this sense, Puppet works pretty much like any configuration file: you say how things should be, and Puppet takes care of the rest.

After the pilot I had the opportunity to train a a dozen other admins with Puppet, plus giving presentations about it in two events. My take from that experience is that some admins took to it, and some didn't. These were all traditional admins, without programming skills, and varying levels of expertise.

One particular thing I noticed is that Puppet requires constant practice. People who were trained, wrote modules, and then spent a whole month or two doing something else came back to Puppet with little useful skill. People who kept doing small things in it every week never lost the ability.

Based on these two observations, I recommend you make sure everyone keeps adding some Puppet class, definition or module every week (preferably at least two or three times). Those who still can't get used to it might really lack the skill to do it.

Then again, if Puppet was imposed on them from higher up, they might simply be reacting to what they perceive as a management intruding in how they do their work -- which would be true enough, in fact. It might be the case that letting them choose which configuration management system to use would improve things. Here are some alternatives:

  • ANSIBLE: This is new, but it's based on shell commands and ssh, which might entice it to traditional sysadmins.
  • Chef: Maybe their problem is declarative style, in which case Chef would be better if they have Ruby experience.
  • SaltStack: Python-based, and open-source
  • CFEngine: old, fast, traditional -- it might win them over on the those grounds.
3
  • 12
    The nice thing about ANSIBLE is that it works across galactic distances with absolutely no delay in data transmission!
    – Kalamane
    Jun 6, 2012 at 20:10
  • 1
    Thank you for the ANSIBLE mention. I wasn't aware of it until now.
    – ewwhite
    Jun 7, 2012 at 0:58
  • @ewwhite You're welcome. I, myself, have only recently discovered it, but much about it has caught my attention. If we hadn't so much in Puppet already, I'd definitely try it out. Jun 8, 2012 at 3:50
11

I've been using Puppet a bit over two years in small shops where I was the only sysadmin. The biggest hurdle I've had is learning how to develop software properly. There wasn't a week that had gone by where I didn't screw up something that I've told developers not to do a dozen times. I checked too much code in, I didn't break up checkins, I didn't tag, I didn't branch, didn't run the syntax checker, didn't use a standard, etc. If you're just starting out I'd recommend some of the following.

  1. Realize you're developing software which you either don't know how to do or are doing poorly. This is expected because it's new.
  2. Infrastructure as code is the reality and once you get over the hump it's quite powerful. I'd invite some developers over, show them your current development process (or lack thereof), don't take offense when they raise eyebrows, and take their suggests seriously. I'd recommend using whatever system and process your developers use unless it's completely inappropriate.
  3. Puppet third party modules suck 90% of the time. I'd read them. I'd steal ideas from them. I would not pull them into my system without major edits. However I would pull in the puppet stdlib which adds some nice functionality.
  4. augeas and hiera. Learn those two. The first allows complex editing of existing files in place. The second is an external data store.
  5. Separate code from data. This is one of the harder concepts to learn. Hardcoding values like Monitoring Hosts into your module code is bad. Putting them in a data store (db, yaml (Hiera uses this be default), csv, whatever) that your modules can consume is good. An example is a webapp that uses Mysql. What this allows is the ability to push code and data separately. This makes your development process simpler.
  6. puppet parser validate and puppet-lint as part of you pre or post code checkin process. Also rspec tests might be a good idea once you're up to speed.
  7. write a style guide/code standard and use it. "where is the code that installs Apache" is a common problem. If your modules are mostly the same it should be easy.

In summary I've hit all of these problems and so have most of my sysadmin friends. It'll take some time to get good at using a config management system. Once you do you'll wonder how you ever lived without one. "Log into a server and make changes manually? Ick."

1
  • Thank you for your suggestions, especially augeas and hiera are two components we have started to implement and this made us much more aware, even confident of Puppet's capabilities. So thanks :-)
    – drumfire
    Jul 18, 2012 at 0:14
7

Six months ago, in our not-for-profit project we decided to start migrating our system management to a Puppet-controlled environment because we are expecting our number of servers to grow substantially between now and a year from now.

Sounds like a darn good idea to start early - Puppet is more than just config management, it's a form of documentation.

Since the decision has been made our IT guys have become a bit too annoyed a bit too often.

They need an attitude adjustment.

"We're not programmers, we're sysadmins";

Again, attitude. You -can- make a conf file for a server right? You can ease into the templating/'programmer' stuff as your needs and complexity evolves.

Modules are available online but many differ from one another; wheels are being reinvented too often, how do you decide which one fits the bill;

Tough one to answer - I always prefer the puppetlabs modules over most - and even at that, I don't use that many. Judgement call for sure. In my opinion, some modules are 'too frilly'.

Code in our repo is not transparent enough, to find how something works they have to recurse through manifests and modules they might have even written themselves a while ago;

This doesn't sound like a puppet problem, but more so an organizational or documentation issue?

One new daemon requires writing a new module, conventions have to be similar to other modules, a difficult process;

That daemon could be a class if it's simple enough to manage. I'm not sure what you mean by conventions, puppet enforces conventions on you pretty well doesn't it? Or are we talking along the lines of code formatting?

"Let's just run it and see how it works"

Not that bad of an idea if you take it slow and safe. I'd still start with a VM to get the gist of things.

Tons of hardly known 'extensions' in community modules: 'trocla', 'augeas', 'hiera'... how can our sysadmins keep track?

postfix, exim, sendmail, mysql, postgresql, iftop, iptraf, perl, perl modules.. Pick what you want and use it? I guess this sound more like an attitude thing again...

I can see why a large organisation would dispatch their sysadmins to Puppet courses to become Puppet masters. But how would smaller players get to learn Puppet to a professional level if they do not go to courses and basically learn it via their browser and editor?

I haven't attended any courses - while I am a programmer more than a sysadmin, I found it didn't need much programming skill to get anything accomplished.

The Puppet documentation, when followed is quite thorough. Just pay attention to the built-in types and spend some time looking at how other modules are put together. I wouldn't say it's -super- easy, but it's not -hard- either. It is a bit time consuming to get your infrastructure ready for puppet, but the time invested is assured to be well spent when you expand.

4
  • FYI this is coming from someone who -just- finished getting their infrastructure ready to roll. So I have a fresh experience and can't say it was time wasted.
    – thinice
    Jun 6, 2012 at 19:05
  • As a recent starter myself, I recognize myself entirely in your comment. Jul 30, 2012 at 14:21
  • 1
    In my case, a change in attitude was indeed necessary. Ops love automation and often script things, so it's mostly a matter of using different tools. It's a cool feeling to see your Puppet manifest configure an entire machine or a new service from scratch. The fact that an error can impact multiple machines at once requires getting used to more rigorous testing, which can be annoying but is obviously a good thing. Experiment with Vagrant, rspec-puppet, puppet-lint, Geppetto, Git branches and other free tools and you'll soon discover your favorite workflow. Jul 30, 2012 at 14:34
  • 1
    Working with Puppet also helped me learn Ruby, which has come to replace Bash as my default system tools language. Jul 30, 2012 at 14:36
5

KISS (Keep it simple stupid) - Don't use new technologies just because they are there rather because you have a requirement for them, use the bare minimum that your deployment requires, update as required don't try to keep up with the bleeding edge. If you start with a basic setup and build on that it is easier to pickup as you go, and they shouldn't need a course (are these even available?).

The other area you can look at are your sysadmins. If they can't program as well, then are they advanced enough for a large deployment, where most of the work needs to be scripted whatever tools you use?

3
  • 4
    ... because we are expecting our number of servers to grow substantially between now and a year from now. Requirement? Jun 6, 2012 at 14:02
  • 1
    Really depends on how certain that expectation is and if what you put in place will still be suitable by the time the need actually arises.
    – JamesRyan
    Jun 6, 2012 at 16:05
  • +1 for "use the bare minimum that your deployment requires" - Alot of the puppet issues i've run into stem from trying to make puppet control everything on the system.
    – Sirex
    Jul 16, 2012 at 23:40
5

I work for a non profit as well and was responsible for initially bringing Linux boxes in house and shortly thereafter Puppet for managing them. There are a couple specific things we've done which have really helped get things rolling.

First and foremost I've tried to stay away from the third party modules. The inbuilt tools handle 90% of our management. The biggest third party utility I use is the firewall module. Any custom facts, etc are developed with the whole team involved. We developed a template module and keep the file management, package, services, etc all standardized off this template.

Second, after standardizing on using the inbuilt modules we started using Git and Atlassian's Crucible - free for non profits, by the way - for performing reviews of all configuration changes. This provides the transparency desired.

Third, I automated the setup for Puppet so that new hosts can be added automatically with a default set of options. There are several ways of addressing this. Since I already had a complete Kickstart environment I opted to add a script there.

4

"We're not programmers, we're sysadmins"

My how times have changed, for the worse: a greybeard like me was expected to be a better programmer than professional programmers, or else would have never been able to pass for a system administrator.

Now, we have "system administrators", who are basically Windows desktop users who have at some point converted to Linux and cannot program, and do not find anything whatsoever wrong with that.

The elephant in the room is why management tolerates such a destructive attitude. Destructive to whom or what? To the business and to the infrastructure.

Back to Puppet[, CFEngine, Chef] subject: as soon as one sets a solution like that in, one loses. Everyone loses. Why? Because whoever comes up with the idea is not capable of designing encapsulated configuration management in the form of nice, clean, Kickstart[, JumpStart, Automated Installer, AutoYaST, Ignite-UX, NIM] operating system packages. When you have to use an automated hacking tool like Puppet (or Chef, or CFEngine), it means you lack the wherewithal to design and to implement a process which would, by that same design, enforce completely pristine and lights out managed systems, fully automated and completely non-interactive.

Another important point is, if you have to have Puppet or some such solution to correct someone hacking system or application configuration by hand, that also goes back to not having the experience to design a process, and in that process a framework where configuration is packaged into discrete components. In effect, whoever implements Puppet and the like, has no concept of component owners, releases, configuration management, Capability Maturity Model. This is rapidly developing into a very serious problem in the industry.

Working with Puppet also helped me learn Ruby, which has come to replace Bash as my default system tools language."

Why is Ruby needed, when a comprehensive, end-to-end configuration management can be encapsulated in preinstall, postinstall, preremove and postremove sections of operating system packages, just by using Bourne shell programs, AWK, and sed? That someone would go to the length of learning an esoteric language of Ruby, and a dialect thereof in the context of Puppet, is completely unnecessary. The problem of configuration management is easily solvable (and to wit, has been solved) with shell programs and AWK, and a little sed(1) here and there as a glue.

It's a cool feeling to see your Puppet manifest configure an entire machine or a new service from scratch.

It is an even cooler thing seeing it done by Kickstart, AutoYaST, or JumpStart, without a single line of code, and being able to query the operating system by using built in tools, without needing any esoteric or extra software, no client-server architecture required (SSH is more than fine, way more than fine), and seeing your operating system being aware of each and every change made to it.

5.Separate code from data. This is one of the harder concepts to learn. Hardcoding values like Monitoring Hosts into your module code is bad. Putting them in a data store (db, yaml (Hiera uses this be default), csv, whatever) that your modules can consume is good. An example is a webapp that uses Mysql. What this allows is the ability to push code and data separately. This makes your development process simpler.

...Or you could just template your configuration files with shell variables, even backquotes (for example ls -1 ...) and write a shell script which uses AWK to call eval(1) and expand all the variables in the template, thereby leveraging the exact same powerful parser which shells have built-in. Why make it complex, when it can be really, really simple? Where will you store the configuration values? Why, anywhere you please, such as for example pkginfo(4) files, or a database like Oracle, or pretty much anywhere. No need for ultracomplex solutions. The library I mention above could simply be sourced from the preinstall or postinstall sections in the operating system packages, thereby removing duplication and leveraging a central piece of code...

But above all, I find that the above quote is an example of the next generation of system administrators needing tutoring not by system administrators, but by system engineers. Find yourself a greybeard and sign on as an apprentice.

2
  • 1
    You seem to have forgotton your answer to the authors question.
    – M. Glatki
    Dec 28, 2016 at 11:33
  • This answer seems to be primarily a discussion about opinions, attitudes and tools, and does not really address the question being asked. Jul 17, 2017 at 15:16

Not the answer you're looking for? Browse other questions tagged .