2

I have multiple webservers which use many small files to created dynamic web pages. Caching the web pages isn't an option. The webserver also performs writes so I need a synchronous filesystem.

I'm looking to maximise performance as it's my understanding that small files is the weakness (to varying degreess) of a cluster filesystem over ethernet.

Currently I'm using Centos 5.5, 64 bit.

Since it's only about 300MB of data, I'm looking at mdadm using RAID-1 with the GNBD and a local hard disk using the "--write-mostly" option so the reads are done using the local hard disk.

Is this possible?

If so, is there any advantage to making it a tmpfs disk instead of a local hard disk?

Or will the files on the local hard disk just get cached in RAM anyway so I won't see a performance gain by using tmpfs, assuming there's enough RAM available?

Improve this question
1
  • 1
    Is NFS not an option? If it is only 300M of data I'd look into using memcache: memcached.org.
    – Mark Wagner
    Sep 27, 2010 at 22:27

5 Answers 5

Reset to default
1

I suggest you look at glusterfs. I use it for 1) transparency - it's backing store, if you will, is an ordinary file-system such as ext3; 2) data availability - glusterfs provides both striping, replication, or any combination; 3) performance and reliability and 4) easy of use.

While you could use it in a (web-server) client / (file-server) server mode, depending on the speed of your network, it could make more sense to me to enable it on each machine. In a sense the file-server becomes the definitive source. Each web-server reads and writes to it's own local glusterfs server, or at the very least it's own cache at local I/O speeds and to the file-server at network speeds making the system quite fast.

It can use tcp or Infiniband. And it seems that it works under Amazon Web Services. It also exports NFS and CIFS so it can be rather portable. Install via yum under CentOS, up and running in under 20 minutes. Compared to GNBD, it is much easier to setup and use. Glusterfs is configured in a highly modular way so you can use only what you need.

The beauty of glusterfs is that it's very tolerant of network or host outages. At my business, whcreative.com, I use it for partially mobile laptops serving home directories as well as html and database file-systems (for the Drupal CMS) in a mixed environment with CentOS 5.5, Fedora 13, and other assorted Linux flavours. Home directories are served from every laptop as well as the server. When a laptop reconnects after being used off-network, a simple >ls -Rl on the server syncs everything. If a machine crashes and the ext4 filesystem potentially has stale data, it's not a problem as syncing to the crashed machine once it's alive solves the problem rather quickly.

The first drawback is it is only tested on x86_64 (claimed to run on i386). Not a big issue though for most. The bigger drawback is it's documentation. For example, there is no man page describing one of the key commands, glusterfs-volgen and the 'man like' page on the website does not provide a working synopsis although it does provide examples. Configuration options are not clearly documented and take a bit of hacking to figure out. The last drawback is that it essentially relies only on user permissions for security. But in the *nix tradition, it is quite easy to run inside a VPN so that's not really a big issue.

I can not vouch for it's reliability as I have only been using it a few months. However, it seems to handle our home directories just fine after disconnecting, using Laptop, and reconnecting. Of course I don't trust it completely and do tar based backups to a CentOS, ext3 filesystem.

Best of luck, Eric Chowanski

Improve this answer
0

RAID-1 will only increase read performance, not write performance. /tmp gets wiped anyways, so use tmpfs on /tmp. Also if you don't care about redundancy, use RAID-0 (striped) and specify a low block size, since you have small files.

Improve this answer
1
  • Write performance is not an issue, it's all about the read performance. After much more Googling, it looks like I can use NBD, GNBD or iSCI to achieve what I want. Leaning towards iSCSI formatted as GFS2. Software RAID1 (mdadm) iSCSI and local hard disk or tmpfs drive should give me the performance I'm after.
    – Travis
    Jul 18, 2010 at 16:50
0

I'm amazed that you consider caching not to be an option with only 300M of data. You can fit that in a $50 stick of RAM (and that's assuming you're not looking for something cheap)

Something else you can consider is adding a RAM disk to the RAID array. You can have as many devices in a RAID 1 array as you like, so you could have a RAM disk, local physical disk, and the GNDB in a RAID 1.

It isn't clear what your requirements are, however. You need high read and write performance, you have multiple web servers. Are the web servers linked in any way? Why is one of the disks in your array over the network if you need performance primarily?

Improve this answer
6
  • LOL, I didn't explain myself properly. I meant the web page itself can't be cached. i.e. it has to remain a dynamic web page. I absolutely intend to cache all the data in a RAM drive as part of the RAID1 array. It is high read, low write. The web servers are setup in a load balanced web cluster. They serve the same files but they also write so that's why I need synchronous writes and hence the iSCSI. I'm thinking, iSCSI in a software RAID1 array with a RAM drive on each webserver. Setup RAID1 to read from the RAM drive. This equals performance and synchrounous writes that I need. A better way?
    – Travis
    Jul 18, 2010 at 18:45
  • OH and iSCSI formatted with GFS2 of course. I'm assuming this can be done but I'm open to suggestions.
    – Travis
    Jul 18, 2010 at 18:47
  • 1
    If more than one machine writes, RAID1 will not work for you. You might want to split the writing responsibility between the machines, and have one writeable partition for each machine, with the others being read-only. Still looks like you want a clustering approach.
    – Slartibartfast
    Jul 19, 2010 at 1:49
  • After some more thought, I agree, it won't work. So how do I keep a fileserver and each webserver's RAM drives in sync? I've looked at realtime two way synchronization for linux but I can't find anything.
    – Travis
    Jul 19, 2010 at 9:05
  • I just fully understood the logic and implications behind you saying to have a writeable partition. i.e. separate the reads from the writes. Might just do what I need.
    – Travis
    Jul 19, 2010 at 9:44
0

To answer the last part of your question, no, I don't think that you'll need to use a tmpfs. Absent other demands for the memory, linux will keep the files in its page cache after they're read for the first time.

The real questions here seems to be "I have a set of files, frequently read but infrequently written to, that I need to keep synchronized between machines. How can I do that?"

The exsiting suggestions seem to be

  • A shared disk filesystem (e.g. an iscsi volume mounted on all the servers, accessed via GFS2)
  • A distributed filesystem (e.g. glusterfs)
  • An NFS server

Of those ideas, I prefer the simplicity of the NFS server. I'll throw out another one- use lsyncd (or some other tool based on inotify) to trigger synchronization when a file is added. Lsyncd makes it easy to perform the synchronization via rsync, but unfortunately this is not a good fit for synchronizing in multiple directions. csync2 is a tool built for this and it can be integrated with lsyncd.

Improve this answer
0

Is Software Raid1 Using mdadm with a Local Hard Disk and GNDB Possible?

The answer: 42It's possible for any block device PERIOD

Improve this answer

You must log in to answer this question.