Edit - I'm starting to look into Perl's Template-Toolkit. It seems to fit the bill nicely for the problem described below. Has anyone played with it? Any (detailed) advice?

I asked a question about config management, and haven't seen a reply. It's possible my question was too vague, so let's get down to brass tacks. Here's the process we follow when onboarding a new customer instance into our hosted application : how would you manage this? I'm leaning towards a Perl script to populate templates to generate shell scripts, config files, XML config files, etc. Looking briefly at CFengine and Chef, it seems like they're not going to reduce the amount of work, because I'd still have to manually specify all of the changes/edits within the tool. Doesn't seem to be much of a gain over touching the config files directly.

  1. We add a stanza to the main config file for the core (3rd-party) application. This stanza has values that
    • defines the instance (customer) name
    • the TCP listener port for this instance (not one currently used)
    • the DB2 database name (serial numeric identifier, already exists, they get prestaged for us by the DBAs)
    • three sub-config files, by name - they need to be created from 3 templates and be named after the instance
  2. The sub-config files define:
    • The filepath for the DB2 volumes
    • The filepath for the storage of objects
    • The filepath for just one of the DB2 volumes (yes, redundant to the first item.
  3. We run some application commands, start the instance
  4. We do some LDAP thingies (make an OU for the instance, etc.)
  5. We add a stanza to the config file for our security listener that acts as a passthrough to LDAP
    • instance name
    • LDAP OU
    • TCP port for instance
    • DB2 database name
  6. We restart the security listener (off-hours), change the main config file from item 1, stop and restart the instance. It is now authenticating via LDAP.
  7. We add the stop and start commands for this instance to the HA failover scripts.
  8. We import an XML config file into the instance that defines things for the actual application for the customer - user names, groups, permissions, and business rules. The XML is supplied by the implementation team.

Now, we configure the dataloading application

  1. We add a stanza to the existing top-level config file that points to a new customer-level config file. The new customer-level config file includes:
    • the instance (customer) name
    • the DB2 database name
    • arbitrary number of sub-config files, by name
  2. Each of the sub-config files defines:
    • filepaths to the directories for ingestion, feedback, backup, and failure
    • those filepaths have a common path to a customer-specific folder, and then one folder for each sub-config file
  3. Each of those filepaths needs to be created
  4. We need to add this customer instance to our monitoring scripts that confirm the proper processes are running and can be logged into. Of course, those monitoring config files include the instance name, the TCP port, the DB2 database name, etc.
  5. There's also a reporting application that needs to be configured for the new instance. You get the idea.
  6. There's also XML that is loaded into WAS by the middleware team. We give them the values for them to plug into the XML - they could very easily hand us the template and we could give them back completed XML.
  • Ingestion? When did that term become en vogue? – jmucchiello Apr 14 '10 at 17:41
  • @jmucchiello - the application is a document repository. "ingestion" is a standard term around here for all of our archiving applications. – mfinni Apr 14 '10 at 20:28
  • I never heard the term (used with data) until a few days ago and I'm wondering how old it is. (I don't usually deal in large archives of data.) – jmucchiello Apr 16 '10 at 17:04
  • At my company, it's been in use for years. – mfinni Apr 16 '10 at 17:45

I think you could use Bcfg2 very successfully for this. It is a configuration management system that is incredibly flexible and extendable. It comes with Genshi for basic templating logic, but allows arbitrary inline Python code to do more complicated stuff.

You could have a single database that contains all information about every instance, and then set up Bcfg2 to generate every bit of configuration based on that. As an added bonus, if you ever have to change something across a large number of instances (or all instances), it will be totally painless.

| improve this answer | |
  • Thanks, I'll start looking into that. We actually have a MySQL database that contains most of the variables I would care about, we're probably 80% of the way there. I know the other 20% will take the other 80% of the time, of course. – mfinni Mar 18 '10 at 20:30
  • I'm probably going to not use that - most of my experience is with Perl, and Template Toolkit can also use a SQL backend. However, you gave the best answer, so you've earned the points. Thanks. – mfinni Mar 22 '10 at 16:06

This looks very custom, so I would look into making my own tool with Fabric for the backend to run all this and Django for the web frontend.

With fabric as backend you can also use it in ssh without django.

| improve this answer | |
  • That really doesn't look very well-suited for the case I've described, unless I'm missing something in the Overview and tutorial. I don't really want to make my own tool from parts that low-level. The problem isn't needing to run scripts on a farm of servers, it's managing a whole bunch of config files to standards. – mfinni Mar 18 '10 at 17:23
  • oh go for puppet then :) – Espennilsen Mar 18 '10 at 18:26
  • Espennilsen - did you read my writeup here? There's not much to be gained if I have to update dozens of Puppet config files / chef recipes, as opposed to directly touching the config files on the server itself. From my reading of puppet/chef/etc, that's what I'd be looking at. – mfinni Mar 18 '10 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.