I have two servers, placed at datacenters in holland and france. Both are running Debian Wheezy. I need to share /home between them, with good performance. There are 300-something users on the servers, around 30 of them should be able to have active processes on a given server at a given time, each having 50 kbit reads and and 20 kbit writes/second, with short peaks around 2000 kbit/s reading. measures with iotop on local storage. I have a lot of small files, around 500000 in total and need as low latency as possible. Ping between servers are 17 ms, and the connection is able to reach around 20-30 MB/s when using scp and wget. It seems there should be plenty of bandwidth available for it too work, but...
What I've trued so far: sshfs: Seemed like it had better performance than nfs, but it ranomly changed permissions of files to root, making the application crash.
nfs: Way to slow, tried noatime an a bunch of other options, but it keeps acting sluggish, even when only a few processes are active.
drbd: 5 hours of dead-end work, when I realized I couldn't actually mount the filesystem on both systems :-(
glusterfs: Having a local copy of all data really sounded promising, but random file access is really slow and after running a while, it becomes unbelievable slow and almost hangs. noatime doesn't help.
nfs again: Still sluggish.
Crying into the keyboard: No improvement at all.
What to try next? Each of the failed trials have taken an evening or maybe more during the last week, and I'd really like the next method to work. And yes, it crucial that the filesystems are shared between both servers.
Thanks for any new ideas on this problem.