2

I have a rather large application that requires simultaneous access to rather a lot (300+) of large files, and accesses them by reading them all at the same time.

If all the files are put on 1 disk (if I have a disk that's even big enough), the execution speed of the application grinds to a crawl. To get around this I distribute the files across a couple of physical disks, and machines, NFS mount the partitions and use bash scripts to create a lot of sym links so that all the files appear to be in 1 huge directory. This works wonders for improving the execution speed, splitting across 3 disks improves execution speed, by a factor of 10 or more.

Does anyone have any suggestions on a less cumbersome way to set this up? My fear is that striping as in RAID or Gluster may not work terribly well, since high read speeds on any one file are never required, but large numbers of simultaneous accesses are always required.

Improve this question

1 Answer 1

Reset to default
3
  • Using NFS instead of many local disks will reduce your speed significantly as any modern disk can saturate a 1GB ethernet connection, let alone more than one.
  • Using a RAID0 will also improve your speed significantly without the administrative overhead, but it will, on theoretical average, be only about 50% of the speed improvement of as many disks with the symlink layout (50% because your performance depends on how many stripes can be used at the same time - in the worst case your app has to read everything from only one disk, in the best case it can read from every single disk).
  • Using SSDs will dramatically improve your performance. It's expensive, but nothing can beat an SSD in this scenario.

So, in short, I would put the disks in the local system and experiment with different RAID chunk sizes to optimize performance and seriously consider to buy SSDs instead.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .