I'm learning chef and having problems structuring everything to work with my team.

For starters, it seems that you should create a chef-repo folder, where you will store and modify the cookbooks used to manage your nodes.

I work on various projects, and each one of them is under git source control already. Ideally I would keep one chef-repo folder in each one of my projects with that projects cookbooks right?

However, in the chef-repo folder I have to add a configuration folder (.chef) with my knife configuration and my key validations, and these are specific to me. Is it normal to just add the .chef folder to the gitignore file?

I understand the cookbooks get uploaded to the chef server to then be deployed. How do other teams separate the staging from production environments without duplicating a lot of work? We have a master branch which is our production branch, a dev branch which is our staging branch (receives less than 5% of the website's requests) and feature branches. Most of the time, the dev branch when stable is merged to the master branch. How can we upload the cookbooks separately to be able to have two environments in a separate way?

Thanks for the help!

  • Could you set up the .chef folder to use environment variables or something? – ceejayoz Dec 29 '11 at 19:31
  • Could you please describe your goal in more detail? What kind of infrastructure do you want to automate using Chef? You've mentioned different branches -- if you're talking about software deployment, a CI tool like Jenkins could be the the better solution. – geewiz Dec 29 '11 at 19:51
  • It's so good how puppet supports environments like this, natively. – Tom O'Connor Dec 29 '11 at 23:03

I work with multiple projects, so cjc's solution won't work for me. There's also an issue of common vs custom configuration (addresses etc are common to the company, there's also a bit of magic in the configs). The scheme I finally settled on is a bit of a hack, but it's a convenient-to-use one.

Instead of global ~/.chef, I use '.chef' subdirectory within chef-repo, which is not stored in git (it's added to .gitignore). I have also file config/knife.rb file which is checked into Git and contains shared configuration. It starts with this snippet:

root_dir = File.join(File.dirname(__FILE__), '..')
%w(knife-secrets.rb knife-local.rb).each do |conf_name|
  conf = File.join(root_dir, ".chef", conf_name)
  Kernel::load(conf) if File.exists? conf

This loads files .chef/knife-local.rb containing custom configuration (in basic version it's just OPSCODE_USER='username' constant that is used later on, but it can contain any knife config), and .chef/knife-secrets.rb which contains shared secrets (AWS keys etc).

Below that, there is regular knife config that uses constants defined in these files, e.g.:

client_key               "#{root_dir}/.chef/#{OPSCODE_USER}.pem"

This way, I achieve standarization of knife config across the company, which in turn means that any code snippet or knife invocation shared in a wiki will work for everyone. There is enough confusion and magic in knife itself - different configurations would only make it worse. Also, everyone gets benefit of small magic snippets, like this one to make knife ssh use login configured in user's ~/.ssh/config

There's also issue of shared secrets: chef server's validation key, AWS keys stored in knife-secrets.rb, EC2's SSH private key, encrypted data bag keys, and so on. We definitely don't want them to be stored in the repository - or, actually, anywhere where they aren't safely encrypted. So we distribute those files as a .tar.gz file, which is GPG-encrypted to everyone in the company, and shared over Dropbox.

Configuring all this is getting complicated, and I want people on team to actually use the thing, so there's the final element: rake init task which creates .chef directory, symlinks config/knife.rb there, decrypts and untars chef-secrets.tgz file, makes sure user's private Opscode Platform key is there and .chef/knife-local.rb is properly configured, symlinks knife plugins, and sets proper permissions on the directory and files inside. This task is set up so that it's safe to run it many times on already initialized repository (e.g. to update secrets or knife plugins).

There's also a helper task that repacks all the secrets, encrypts the tarball to everyone, and copies it into dropbox, to make it easier to add new employees or change secrets.

Regarding multiple environments: Chef has a feature called environments. I haven't used it yet, but it should do what you need. You can also strictly separate production environment (to avoid developers having any keys that relate in any way to production env) by having two separate Hosted Chef organizations or Chef servers. This knife.rb snippet show how to configure knife in a different way based on currently checked out branch - you can use this to set environment as well as chef server's url. There's also knife plugin called knife-flow , providing more complete, two-organization workflow.

| improve this answer | |
  • Thanks for the detailed answer. Seeing the work that goes into setting it up at least means I did not miss something obvious... – Alex Recarey Jan 2 '12 at 14:34

You need to set up two Chef servers, one for production and one for development. The reason is that no single Chef server can support branched development; even with environments.

Or you can abandon the Chef server concept and use chef-solo. You can maintain your cookbooks in Git. You can branch & merge. You can ignore the problems with knife credentials because you won't use them any more.

You won't be able to use knife search or data bags**. But some people don't need those features anyway.

** Well you sort-of can: http://wiki.opscode.com/display/chef/Data+Bags#DataBags-UsingDataBagswithChefSolo

| improve this answer | |

I have two directories in my home, .chef and chef-repo. The chef-repo is in git. The .chef is some private directory that's the default for knife. You don't have to put your secrets from .chef into git; knife will look for ~/.chef.

| improve this answer | |

Your ~/.chef directory should not be in the git repo.

I have a ~/projects/ directory under which I keep my chef repo. In this goes my servers' configs.

My last job was as a Systems Engineer at at Ruby-on-Rails shop. Our nginx, varnish, and rails configs (among others) go in the chef repo, but the Rails apps themselves were kept in separate git repos, and were deployed separately.

Our Staging environment was a single server that ran the entire staging environment. This was not ideal because it didn't resemble Production where Rails thins and the DB were on separate boxes. What I would recommend is using Chef's Environments to separate Staging and Production. (That's how it was when I got there, and I just disn't have the time to fix that before I left.)

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.