5

I always worked where they used to run Puppet at regular intervals. So, distributing changes was easy and on the fly. In the new team, they frown upon running the Chef agent at regular intervals. They only use it to bootstrap the OS and then kill it. I don't understand why would anyone use a config management tool like Chef without having to run it regularly. Whatever bootstrapping we are doing could be done via basic shell scripts - Install xyz software, copy config file, restart the service.

They say that its too dangerous running it at regular intervals in production as they aren't sure if the code is idempotent.

My queries are:

  • How many of you use the Orchestration tools just for bootstrapping? Isn't it like driving a Bugatti at 20mph in alleys?
  • Are there any problems you see in running this at regular intervals, when you scale up? How would you handle it? (One way I know is to run the agent's in solo mode and let them download the cookbooks from some repository/artifactory that can handle simultaneous multiple downloads, rather than overwhelm the Puppet/Chef server).
  • How can I encourage the team to fix the code to being idempotent and run the agent at regular intervals? Or move away from Chef to something simple as bash to reduce the overhead of maintaining/writing the code.
  • Am I right in saying, that we are not using the tools the way they are supposed to be used?
  • Am I missing/overlooking anything here?
4
  • Does the team have any test bed, change management and continuous integration process in place? I think the issue seems to be about workflow and organization not technical. I do agree that there is a danger of deploying on the fly however one needs to look at pros and cons. Is bootstrapping causing delays in deployment? What would you gain if you do this on the fly? Jul 6, 2018 at 8:24
  • We do have Dev, Stage envs in place where we usually test our code. What kind of danger are we talking about? That's the question I posted. Isn't it the same risk when you deploy an app in production? You test your code in Dev, stage and then deploy it in prod. I don't understand how this is any different. No, bootstrapping isn't causing delays. Why do you ask? Doing it on the fly let's you keep your environment updated. Now we have no way of upgrading that openssl package on the fly or update the configuration. All changes go into pending list. There's no way of distributing those changes.
    – deppfx
    Jul 6, 2018 at 16:54
  • 1
    It's more like crashing your new Ferrari right outside the dealership. Jul 9, 2018 at 23:41
  • @MichaelHampton True. That's why don't drive a Ferrari you don't deserve. :-)
    – deppfx
    Jul 10, 2018 at 0:45

1 Answer 1

3

orchestration for bootstrapping

There are tools like Terraform which are actually focused on this part of the process. I also use ansible for some ad-hoc tasks that don't need to be rerun often.

Generally though, it is a best practice to run your config management at least every hour. Granting or removing access often happens via these mechanisms and delaying updates can cause compliance or usability issues. At one large shop we split puppet into two so the app-specific stuff could be paused without breaking the "shadow puppet" which handled access control updates and "couldn't" be cut off.

problems from running regularly

If you write bad recipes then you can destroy all of production very quickly. Having some process where roles are released into QA and validated before going to staging and being re-validated before going to prod. Chef has built-in testing mechanisms. Similar techniques can be used with the others.

how to encourage running it regularly

I would first focus on the problems that are being brushed under the carpet. If you don't run your recipes often then you won't notice when they start to not work because of changes to the OS or your apps.

Then I would mention that changes can be made everywhere pretty quickly when needed. Your interval between chef runs should be the maximum amount of time you're willing to wait for a change to propagate throughout your environment.

are you right?

Mostly. If it works well enough for them they may not see any need to change anything. You might need to come up with a demo to show the value and make it real for people. Or you might need to wait for your organization to mature to the point that it can handle what you're teaching.

what are you missing?

The main thing you don't seem to be considering is the possible performance impact. If the app is really sensitive to things running in the background then you can see lower throughput or higher latency while chef runs. If this is the case you'll need to adjust your recipes or only let it run at off-peak times.

Another thing I've seen happen is memory exhaustion. The app gradually chews up memory until chef can no longer function. Hopefully you have monitoring of memory levels and whether chef is working or not to catch this sort of thing.

Beyond performance and memory I'd suggest reading a book like Release It that explains a lot about how to build reliable production systems.

1
  • Yes, we have every monitoring in place. Memory exhaustion should be the least of worries as we barely use 50% of it and we do experience occasional CPU spikes. But afaik, that should be totally okay if there are not too many changes pushed at one time. Thanks for your feedback.
    – deppfx
    Jul 10, 2018 at 20:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .