External architects have designed an architecture for a Service-Oriented Architecture environment. They fixed the availability to be 99.9%, equivalent to max 9 hrs downtime [per year].

Where can I find resources on how to map this to a physical architecture?

The main components we think we'll need so far are:

  • A drupal CMS with its database (one machine)

  • APIs: Authorization, Authentication, Logging, Management, PubSub, and a Learning Record Store LRS: (One Application Server machine)

  • One Database server for storage (LRS, Logs, Profiles)

This is for a municipal environment for a 5 million souls city. There's no data on concurrent access requirements and stuff like that, but I'd rather not go high-end on this issue. Strategy is to start lean but robust and expand if needed.

Please you are very welcome to require more information or to comment on the first shot of the component architecture - or on anything you might deem pertinent. Thank you.

  • 1
    Clusters and redundancy on everything; no single points of failure. Or were you looking for something more detailed? That's what consultants are for. – HopelessN00b Mar 13 '14 at 14:05
  • @HopelessN00b or a 500 rep bounty. – MDMoore313 Mar 13 '14 at 14:06
  • @HopelessN00b thank you that's what I need to know basically. What would be great is some resource to which I can point to to justify the requirements of clusters and redundancy based on their defined availability. Does that exist? MDMoore313 don't have 500 rep yet ;) – Fabio Mar 13 '14 at 14:09
  • @Fabio it exists if you have the rep, or a friendly soul willing to put it up on your behalf in a couple of days. – MDMoore313 Mar 13 '14 at 14:11
  • I need rep crowdfunding then... – Fabio Mar 13 '14 at 14:12

99.5 is quite (extremely) crappy and easy to do - remember that a single machine has a uptime of 99.9%, so technically you just need to keep a separate server available, configuration in place, backups every couple of minutes.

With proper setups you should be able to get the reserve machine up withint 10 minutes maximum.

That handles everything besides datacenter outages, but those are SLA level and then you can have something like azure / amazon with the second machine on a second data center.

  • Aaaaah...I see I have a mistake in my description - it's 99.9%, not 99.5% (will edit). Still this would mean your considerations apply. – Fabio Mar 13 '14 at 14:30
  • 1
    @Fabio even then. GIven 99.9 is a normal workstation computer and 9 hours is a lot - the advice stands. Problems start normally with 99.99% -anything less is "Make sure you have spare parts, live backups and fast access to a techncian". Systems do not fail every day and in a data center a technicial should be able to work on for example a dead switch within half an hour. I pull db backups of my main working stuff every 10 minutes and move them to a backup machine. Downtime in case something fails is minimal. 99.9 is really trivial - noone would call that high availability. – TomTom Mar 13 '14 at 14:34
  • Basically: You have plenty of time for getting reserve online, but no time to wait for delivery of spare parts ;) That is what 99.9% means. – TomTom Mar 13 '14 at 14:35
  • thanks for the time you took to deliver detailed information to me! – Fabio Mar 13 '14 at 14:48

I do highly-available Drupal for a living as an engineer at BlackMesh. As TomTom said, anything 99.9% ("three nines") and below is very straightforward.

Getting above 99.9% is easy if you want to spend money adding servers and load balancers and all that jazz. If you don't, though, there might be another way. I'd rearchitect your solution into two machines, and spend the savings from the lack of the third machine on making the other two more powerful. Perhaps counter-intuitively, you can also seriously consider sacrificing some level of redundancy, as well—for example, you might forgo a secondary power supply on a physical machine in favor of simply connecting each of these two machines to different buses.

Anyway, what you do with these two machines is set them up in a proper cluster. Have HA IP addresses that fail back and forth between them.* Machine one is the default Web machine, running Drupal. Machine two is the default DB and LRS machine. In the event of a failure, the addresses move such that the remaining machine takes the full load. For MySQL, this will require master-master replication; for Drupal, it will require syncing of the DocRoot (paying special attention to the "files" directory); for your LRS, it may necessitate a manual failover process.

With this type of configuration, with decent hardware, good power, etc., you should expect to see five nines (99.999%) of OS uptime (which works out to about five minutes of downtime per year), and close to that number in terms of Layer 7 availability. Since you said you'd need to justify the numbers you came up with, five nines presumes a shared-nothing environment, and is simply the failure rate of an individual server (1.0-99.75%==0.25%) squared to represent the probability of two servers going offline at once (1.0-0.25%*0.25%==99.999375%), with a bit of a fudge factor.

Lastly, I should note that these sorts of SLA's are a bit of a red herring. See https://serverfault.com/a/161141/46760 for my thoughts on this matter. The reality is, you'll lose DAYS of functional availability due to someone fat-fingering a click on the Drupal admin account (or something similar) over the life of this solution. Setting up good change control and similar processes should be given equal billing, if not first billing, over hardware redundancy discussions.

*: n.b. There is a potential for split-brain in any cluster with an even number of members. One of the better ways to lower the probability is to route health check traffic over the public interfaces. If you want to be really paranoid, look into STONITH over serial.


Before designing any architecture i'd check your environment: Do you have an internal IT department and what do they host?

If there is a database server already installed it could be able to host your database(s) as well.

What are your sysadmins familiar with? If your a Microsoft shop, a solution based on any *nix could easily upset your admins (or vice versa)

Regarding the downtime, do all parties involved accept that the nine hours downtime only count for unplanned downtime and not do not include downtime for patching and upgrades? How much data needs to be restored? Can the whole system be restored within the nine hours with your backup system?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.