0

Thing that confuses me about configuration management tools such as Puppet is that they're great and straightforward for setting up server from scratch, but what every documentation neglects is the case where I have existing data (e.g. PostgreSQL database) that I want to use on the newly provisioned server.

The perfect example is migration between hosting providers. Sure Puppet will set up the new server with (almost) same configuration as the previous one, but the setup code (Puppet manifests) assumes that this new database server is at its genesis whereas there are precious business data that needs to be restored.

Setting up infrastructure for the first time is surely a task worth automating, but accidents happen and when the need for restoring the server arises, I'm left with semi-manual setup:

  1. Puppet: Setup the server up to the point before data directories are initialized and services are started
  2. Me at shell: Restore data from backups to proper places
  3. Puppet: Continue with the rest, i.e. start the services

This cannot be that uncommon. Setting up servers for the first time with no existing data is one time event but recovering them can happen many times, and I execpt it will happen at least once (that's why we backup precious data).

What's the common practice in writing Puppet manifests to address the scenario that the node can be setup for second time and needs to restore data?

How do you deal with migration of servers managed by Puppet?


Googling this is hopeless. You mostly encounter pages describing how to restore Puppet DB/server but that piece of infrastructure mostly is not what your business generates profits from, right?

I realize that Puppet server has a fileserver, but keeping the backup data available there and setting proper access control feels very tedious to maintain compared to managing configuration with Puppet manifests.


I assume enterprises use full network backup solutions like Bacula or Amanda, or perhaps custom backup solutions based on Borg, Restic, Burp, Duplicati or plain rsync. Do you integrate these into your Puppet manifests? How?

2
  • 3
    Normally, you don't do it that way. You'd have a cluster of database servers, and use puppet to setup a server to be joined to that cluster. You'd only need to do a full recovery when the entire cluster has failed.
    – Jenny D
    Sep 21, 2019 at 14:05
  • @JennyD That's good advice. However, that's specific for database technology used. Some databases do not operate in clusters. Sometimes such high-availability is not needed.
    – woky
    Sep 21, 2019 at 15:26

2 Answers 2

2

If you do this for fully automatic disaster recovery purposes, it might be worthwhile, but otherwise you have to think about if you actually save anything by investing the significant amount of time you need to reliably (!) automate what should be a very rare one-off operation (if you have to do this all the time, something's very wrong in your environment). Having Puppet doesn't mean you have to automate everything at all costs ...

That said: If you need it, it's not difficult: Just have a backup solution that supports doing this from Puppet - this support can be just running a script that does what you would be doing manually otherwise inside a Puppet run (make sure to run it only once...).

2
  • By your argument in first paragraph investing in configuration management solution like Puppet is also waste of time, because initial setup is precisely one-off event. Compared to initial setup, accident or migration may never happen, but it can also happen more than once, and will probably happen at least once, so it's on par with initial setup.
    – woky
    Sep 21, 2019 at 15:22
  • 1
    That's wrong is most environments. Automating a process that you have to do possibly hundreds of times and that saves you potentially hours of work per machine is something very different than writing a reliable restore process for something that a) you should nearly never to in the first place b) requires only a few minutes of actual work from you anyway.
    – Sven
    Sep 21, 2019 at 16:02
2

Using Configuration Management (CM) tooling such as Puppet, Ansible, Chef or Salt needs a fundamental change in the way you think about configuration.

If you think about configuration as a one-time event, you will never see (or realise) any benefit from using CM.

The main power of CM is the ability to express your desired configuration as code. Once you have that ability, your server configuration is:

  • Testable. You can write unit tests to make sure you're not introducing any bugs, all without restarting a service or provisioning an entire test/staging environment. This saves a lot of time and money.

  • Reproducible. You can create one server, two servers, ten servers or a hundred servers using the same base configuration, with no overhead. You write the code once, and apply it many times. If you have a standard set of configuration options (for example, SSH daemon settings, hardening steps) CM can automate those away.

  • Versionable. Rolling out a change is as simple as making a Git commit. Rolling back a change is as simple as git revert. You have a verifiable, audit-able data source (your Git history). You can branch your CM codebase, make changes, get peer review and merge those changes into your master branch.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .