4

If you provide a service on two servers to ensure high-availability, is it better to configure them in exactly the same way, of instead should you introduce slight differences to prevent "freak configuration" errors?

We host a Django-based website on a stack of Linux (Ubuntu LTS), Nginx, Apache and Python WSGI, duplicated on three servers behind a load balancer. Currently they are hosted in the Amazon cloud, but we might move to our own datacenter in the future. We recently had an issue on all three servers which was only solved by upgrading the kernel, which makes us think it was an incompatibility between this specific version of the kernel and the physical hardware that Amazon might have started using at the point.

This made me think: would it be better to keep all machines on exactly the same configuration (easier management?), or should we instead keep things slightly difference, so that an incompatibility between two components will only manifest itself on one machine and not all of them, keeping your website in the air?

Improve this question

3 Answers 3

Reset to default
10

Keep them the same. The chance that you will have some incompatibility that manifests itself only in a certain configuration are minimal and afterwards you will have to remember the differences for everything you do.

Improve this answer
1
  • 1
    definitely. This is why you have staging servers and pre-production servers. Keep the actual server pairs identical.
    – Sirex
    Oct 6, 2011 at 10:13
1

For simplicity they should all be the same config, however there are occassions (mostly dictated by the software in use) where it's just not possible to load-balance and failover becomes the only option - and in such cases having slighlty different configs may be required.

OTOH, for an internet facing service, availability and security must be high on the list of priorities. Good security means applying patches regularly, good availability means that you cannot patch all the boxes at the same time - indeed, the practice I adopted for a similar setup was to apply patches to one live machine as soon as they were available and had been applied and briefly evaluated on a test machine, but to delay the rollout to the other nodes for a couple of days until I knew the patches did not have any adverse effect.

While Sirex is correct - in a perfect world - you would implement patches on a pre-production cluster and test using traffic / data from the production system - in practice this is far from cost effective on such a small scale.

Improve this answer
2
  • in the case of hardware issues it can get expensive to have ideal testbeds, but in our case we use virtual machines for this as software issues and misconfiguration are far more likely.
    – Sirex
    Oct 6, 2011 at 13:17
  • Good point to use some "monitoring days" before applying patches to all the nodes, thank you for that!
    – Jan Fabry
    Oct 6, 2011 at 15:55
1

Yes definitely. This will help with troubleshooting issues that arise.

Look at Puppet to manage your config file changes. We would store config files in svn then push our changes. We had a centralized management server that would check out our changes and Puppet would push them. This gives a history of changes so when you make a mistake you can roll it back pretty seamlessly and when you have multiple admins the config changes can be tracked.

Ref: http://puppetlabs.com/

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .