I'm looking for the best way (for my purposes and situation) to version system configuration files and manage deployment (ownership and mode) between just two machines (live and backup, both should be regarded as 'production'). I would like to be able to use this system to automatically (but selectively) duplicate and manage relevant and specified configuration on the backup machine (an EC2 instance).
Currently, those machines are primarily web and SMTP servers (we don't store user mailboxes), but our project is growing and who knows what future requirements may be. For example, there is the possibility that we'll implement radio and web-based chat at some point in the future.
I prefer to manage the server conventionally (ie, go in and make changes to the live server as required) and then store those changes (once settled) in the management system. (Okay, if I'm honest, I should say organically: I don't want to lose the flexibility to do stuff as needs and situations demand.) I find this approach, rather than deployment of config that was tweaked and tested elsewhere, more predictable and with the potential for fewer surprises (outside of the possibility of screwing up configuration on a live system, that is!)
I realise that, philosophically speaking, this is back-to-front and I probably would do it the other way around if I had the resources (test infra and, more importantly, manpower). As it is, I've got to make do with what I've got. I like structure well enough, but excessive infrastructure takes on a life of its own and at some point the value in it is lost.
I've done some homework and reviewed a couple of options, but none of them seem to really do what I want them to, so I'm currently tempted to roll my own solution. The purpose of this question is to see what I've not thought of and to sanity-check before I go barking up the wrong tree.
Declarative metaconfiguration systems like Puppet, inter alia, are probably good choices where many servers are involved with little or no differentiation between them, and especially when it's clear ahead of time what configuration should look like. I don't think it'd be wise to employ such a tool without staging infrastructure.
It's also a bit heavy-weight and overly complicated. I'm the only admin, I do this in my spare time, and should something happen to me whatever I leave behind needs to be reasonably sane for the lucky guy/girl who steps in after me.
etckeeper might do, but so far as I've seen, it doesn't really lend itself to managing anything other than /etc. ie, it wouldn't be suitable for storing custom scripts which, by tradition, live in /usr/local/bin. Also, etckeeper presumably manages all of /etc, where I think I'd prefer to version only specific stuff (eg postfix, apache, fail2ban, ssh and possibly munin).
A simple git or svn repo would certainly not do, for while git tracks permissions, it doesn't track ownership, and svn tracks neither.
svn keeps a
.svnsubdirectory in every tracked directory, whereas
.gitexists only in the working copy's root directory. There are benefits and disadvantages in both cases. The presence of an
.svndirectory in a certain location might be okay in some places (eg /etc/apache2/sites-enabled) but upset brain-dead scripts/tools elsewhere. (I'm sure I read something a while back: was it that cron was okay with a
.svndir in /etc/cron.d, but depmod was not with /lib/modules?)
On the other hand, with a
.gitin /, git considers everything a candidate for tracking, even (presumably) other git repos!
Here's what I propose to do: a subversion repo stored on the primary machine (say /usr/local/sc-repo) and a management script written in Pysvn that implements the following:
add (but, by default, not recursively), which stores the file's mode and owership as custom properties. Also, I like
$Id$tags in my config files, so this will ensure
svn:keywordsis properly set.
update, which applies ownership and mode after the file is written
perms, like diff, but for ownership and mode, with fix flag to correct if required.
The backup machine will access the repo via ssh and perform an update operation every night. (It will do other things, too, like restore site SQL and application filesystems from the main machine's backup repo (S3), and I will probably have to write some hackery that looks for new packages added on the main machine and adds them on the backup machine — but all that is outside of the scope of this Question.)
I know, like and am much more familiar with svn than I am with git. Yes, I'm probably a dinosaur.
Pysvn is easy, and I've used it before.
svn supports versioned custom properties, where git doesn't seem to
this is not distributed development: distributed repos complicate matters without benefit. Remote repo availability is irrelevant, because commits from the remote machine will be rare.
I'd be grateful for your feedback. I have thought as much of this through as I could, but I'm bound to have missed something.