I have a rather large application that requires simultaneous access to rather a lot (300+) of large files, and accesses them by reading them all at the same time.
If all the files are put on 1 disk (if I have a disk that's even big enough), the execution speed of the application grinds to a crawl. To get around this I distribute the files across a couple of physical disks, and machines, NFS mount the partitions and use bash scripts to create a lot of sym links so that all the files appear to be in 1 huge directory. This works wonders for improving the execution speed, splitting across 3 disks improves execution speed, by a factor of 10 or more.
Does anyone have any suggestions on a less cumbersome way to set this up? My fear is that striping as in RAID or Gluster may not work terribly well, since high read speeds on any one file are never required, but large numbers of simultaneous accesses are always required.