2

We're looking for a way to improve the reliability of one of our servers (Apache/MySQL/Virtualmin setup). So far, we've had every possible kind of clusterfuck with that server in the past six months (DNS failure, DDOS, Dom0 failure, network outage, DomU failure, …; on a good day 2 at once) and while all were resolved in less than a day, it's still worrisome – there are about 50 customer websites on that host, and they will get on our throats every time the server is down (the server's availability is still over the contractually guaranteed 99%, but well… you remember the 5 occasions the server had downtimes, not 360 days it was up).

Plans so far:

  1. Backup DNS server (shouldn't be much of a problem)
  2. High-Availability setup for the server itself. The problem here is data replication to the secondary host.

The hosts would be in different (Hetzner, btw.) data centers, so we'd have a rather limited bandwidth (100 MBit uplink, and there should be at least some bandwidth left for the actual users…) and data encryption is more or less a fixed requirement.

DRBD itself scales poor over WAN, neither does it provide encryption. DRBD proxy claims to solve the bandwidth problem (but not the encryption problem, as far as I can see), but it's simply too expensive from what I read, $5k/year are too much (I'm pretty sure that's more than what we're earning with that server).

On the other hand, from my personal experience, OpenVPN/SSH tunnels are not reliable enough to guarantee we wouldn't have false alerts triggering unnecessary failovers (never mind the overhead reducing hard disk performance even more).

So... what alternatives are there? Or am I simply overlooking something?

Edit: To clarify, I'd prefer a replication on file-system/block-device level. Application-level replication is possible, but I'd rather have one replication solution running than one for each application.

Improve this question
7
  • 8
    Cheap, Fast, Reliable. Choose Two.
    – Tom O'Connor
    Jan 10, 2012 at 13:57
  • what about glusterfs ? I've not used it myself but i heard it works better over wan links than drbd would.
    – Sirex
    Jan 10, 2012 at 14:03
  • @TomO'Connor: I know, I know. I still want to try it. ;)
    – Creshal
    Jan 10, 2012 at 14:15
  • @Sirex: Do you have any benchmarks? And know about encryption/compression? I'll have a look into it anyway, thanks.
    – Creshal
    Jan 10, 2012 at 14:17
  • 1
    @ChrisS Fast = Fast to operate (and Fast to implement)
    – Tom O'Connor
    Jan 10, 2012 at 15:35

1 Answer 1

Reset to default
5

You seem somewhat stuck with the DRBD replication. I would think this is because it does not suit your needs. It replicates block devices and is quite bandwidth-intensive (although link compression might alleviate that quite a bit). Check out if you would not be happier with replication at a higher level - like MySQL replication mechanisms for the databases and something like lsyncd for the filesystems.

Gluing it together with stuff from the linux-ha project or setting up a semiautomatic or manual failover mechanism in conjunction with some monitoring is surely a bit of work, but should give you what you want in the long run.

Of course, you still would need an encrypted tunnel for the traffic, but I do not understand your reluctance to using OpenVPN - as the tunnel is just there for the sake of a backup/standby system and you would have either a witness (in a HA setup with automatic failover) or a monitoring system (in a setup with monitoring) which is independent from the presence of the tunnel, you would not have any failovers on tunnel outages and just get the alarms to fix the tunnel upon outage (which is of course necessary, otherwise you lose the capability to do a failover to an up-to-date standby system).

Improve this answer
5
  • Monitoring isn't the issue, that's already sorted out (linux-ha was my choice, yes). The only remaining issue was data replication – I was originally looking at DRBD because we're using it already for other setups (and I wouldn't need to familiarize myself with a new solution). Though I would like to have a file-system level replication – yes, replication on application level is possible, but needs to be done for every application. Tunnel outages would be headaches with DRBD, and I'd rather have a solution that doesn't need fixing than one that does. :)
    – Creshal
    Jan 10, 2012 at 14:30
  • Although, yeah, as long as the replication is fine with outages, I could resort to OpenVPN.
    – Creshal
    Jan 10, 2012 at 14:33
  • 1
    DRBD is capable of asynchronous replication as well, but bandwidth requirements would stay the same and you always end up with an unclean filesystem on failover which is likely to cost time. Replication at the filesystem level has its own problems, but in a typical "my webserver is generating a lot of logs and temporary data but only few persistent files" scenario it should work well enough.
    – the-wabbit
    Jan 10, 2012 at 14:42
  • It's mainly a webserver. Logs are already sent over the 'net (as far as they're using syslog), activity is mainly database and your odd git pull. So, lsyncd for the non-sql-storage partitions and MySQL replication should suffice and have acceptable performance and reliability? A few minutes missing aren't that bad, as long as the data is consistent.
    – Creshal
    Jan 10, 2012 at 14:50
  • The system is now running like a charm with lsyncd (+MySQL replication). Thanks for pointing me to it!
    – Creshal
    Feb 21, 2012 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.