0

I am planing to move my small shared-hosting environment to Amazon EC2. It requires a central storage for the webstuff (leave databases and so on aside for now) so that i can do both: load balance a website across multiple http nodes and/or move a website from one node to another without actually moving any data.

Currently, my storage is on a NFS Server, which is made high available via pacemaker/heartbeat/drbd. Afaik, you cannot work with virtual IPs in EC2, so most HA solutions drop out. What remains is GlusterFS (and Ceph, but they themself state not to use it in prdocution). So i would setup two GlusterFS nodes, each in a separate availability zone, each mounting multiple EBS volumes (mdadm raid 10 for performance and paranoia) and replicating to each other. Each client would be aware of both exporter nodes and if one goes down, the storage itself would be still up.

So far so good (or not?). Now comes the problem: from my understanding of GlusterFS the access controls are volume based hosts.allow/.deny lists. To reduce the impact of a corrupted HTTP node, i'd like to limit the access from each node to the storage to the webroots of the websites he actually serves.

Let me clearify that with an example of my current NFSv4 + mount-bind setup:

On httpnode1

those are the websites he serves

* domain1.tld
* domain5.tld
* domain10.tld

the nfs export in /etc/fstab

10.0.0.1:/ /srv/web nfs4 rw,..

On the NFS exporter

the structure

* /srv/storage   << here are all websites
* /srv/export/httpnode1/   << webroots for httpnode1
* /srv/export/httpnode1/domain1.tld/   << mount-bind of /srv/storage/domain1.tld/
* /srv/export/httpnode1/domain5.tld/   << mount-bind of /srv/storage/domain5.tld/
* /srv/export/httpnode1/domain10.tld/   << mount-bind of /srv/storage/domain10.tld/
* /srv/export/httpnode2/   << webroots for httpnode2
* /srv/export/httpnode2/domain31.tld/   << mount-bind of /srv/storage/domain3.tld/
* /srv/export/httpnode2/domain5.tld/   << mount-bind of /srv/storage/domain5.tld/

/etc/exports

# httpnode1
/srv/export/httpnode1   10.0.0.123(rw,..,fsid=0,crossmnt)
/srv/export/httpnode1/domain1.tld   10.0.0.123(rw,..)
/srv/export/httpnode1/domain5.tld   10.0.0.123(rw,..)
/srv/export/httpnode1/domain10.tld  10.0.0.123(rw,..)

# httpnode2
/srv/export/httpnode2   10.0.0.192(rw,..,fsid=0,crossmnt)
/srv/export/httpnode2/domain3.tld   10.0.0.192(rw,..)
/srv/export/httpnode1/domain5.tld   10.0.0.123(rw,..)

What i can think of, so far, there is only one solution with EC2 + EBS + GlusterFS would be to export multiple volumes from GlusterFS: one per HTTP node.

Does anybody have any experience with this (eg ressource requirements in some hundred exported volumes) and how do look your configs for this? Or even better: Is there any elegant approach to achieve my goals?

Thanks for any insight!

Greets Joerg

1 Answer 1

0

My experiences with gluster in the EC2 were less than remarkable. I saw HUGE performance problems and ultimately ended up yanking gluster out entirely. In short... if there's 1-2ms of latency between the nodes... gluster will suffer exponentially. This was a fairly small setup with only 2 nodes in it. I would hate to see if you had more nodes or shares.

My solution was to switch to unison & replicate between instances... but I can honestly understand that this may not be ideal for many situations.

As an alternative solution, have you considered using the s3 bucket & s3fs? That can be easily shared between nodes.

3
  • Thanks for your reply. Sad to hear about the issues. Hmm, replication will probably not work for me, especially because i would use GlusterFS due to automatic-failover. But i will look into s3fs, could not find any benchmarks so far, but thanks for the hint.. i don't know why i've overlooked this .. do you have any experience on s3fs + http (apache, fcgi issues and son) ? Apr 29, 2011 at 14:19
  • I do have experience with it... and I could make arguments both ways. (use it vs. don't use it) My only complaints with gluster were that it behaved very much like a replicated fs... but only sync'd when a read-action was performed. The down side of this was that every time you accessed a file in one of the directories, you had to wait for it to check every peer for consistency... and wait for replication to complete even BEFORE the file is actually read. This introduced some nasty lag-spikes.
    – TheCompWiz
    Apr 29, 2011 at 14:25
  • Thanks for that, even though it's quite discouraging.. S3fs probably won't work for my use case, cause i still need the separated access (httpnode1 -> only website data served from this node) whilst all website data is on one storage (access from multiple nodes, fast transfer websites between nodes). The former would require an exporter layer in between (eg NFS, GlusterFS), as far as i understand it. Ok, back to the scratch board then ;) Apr 29, 2011 at 14:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .