Thing that confuses me about configuration management tools such as Puppet is that they're great and straightforward for setting up server from scratch, but what every documentation neglects is the case where I have existing data (e.g. PostgreSQL database) that I want to use on the newly provisioned server.
The perfect example is migration between hosting providers. Sure Puppet will set up the new server with (almost) same configuration as the previous one, but the setup code (Puppet manifests) assumes that this new database server is at its genesis whereas there are precious business data that needs to be restored.
Setting up infrastructure for the first time is surely a task worth automating, but accidents happen and when the need for restoring the server arises, I'm left with semi-manual setup:
- Puppet: Setup the server up to the point before data directories are initialized and services are started
- Me at shell: Restore data from backups to proper places
- Puppet: Continue with the rest, i.e. start the services
This cannot be that uncommon. Setting up servers for the first time with no existing data is one time event but recovering them can happen many times, and I execpt it will happen at least once (that's why we backup precious data).
What's the common practice in writing Puppet manifests to address the scenario that the node can be setup for second time and needs to restore data?
How do you deal with migration of servers managed by Puppet?
Googling this is hopeless. You mostly encounter pages describing how to restore Puppet DB/server but that piece of infrastructure mostly is not what your business generates profits from, right?
I realize that Puppet server has a fileserver, but keeping the backup data available there and setting proper access control feels very tedious to maintain compared to managing configuration with Puppet manifests.
I assume enterprises use full network backup solutions like Bacula or Amanda, or perhaps custom backup solutions based on Borg, Restic, Burp, Duplicati or plain rsync. Do you integrate these into your Puppet manifests? How?