Lets say I have a couple of thousand large files (1-800mb each), that are all accessed at random, with newly uploaded files accessed very frequently, and as time passes, the access times drops off in an inverse square fashion, but there might be random spikes in usage of the older files.
The total throughput is in the 2-4Gbit range.
Im looking for a self-hosted solution, not the Amazon offersings, since they are way too expensive.
What I roughly had in mind is this:
Expensive "main" server with several 15k rpm SAS drives (or SSDs) which would be hosting new files that are just uploaded to the site. Once the download rate drops (or file reaches a certain age) its moved to one of the cheaper archive nodes.
EDIT: Files are to be served via HTTP to a wide variety of users. Servers are running FC5. Need mostly read access, but write is important also.
Right now I got a simple 2 server setup maxing a gbit, and Im getting crazy IO. The box is formatted with 4K blocks. Would increasing it to say.... 1024K have a huge impact?