I just installed an LSI 9260-i8, using two virtual drives, the first composed of 4 SSDs, the second of 4 HDDs. Obviously the idea is to get better performance while maintaining some security and plenty of storage capacity.
The SSDs are great and that array is ridiculously fast when dealing with small to relatively large files. The HDDs host mostly huge files (500MB-30GB). It's intended a the main long term storage facility, while the SSD array is for operating files and short term storage only. This means very often files will be moved from the SSD array to the HDD array.
Problem is that performance very quickly declines after the first gig or so of a large operation is written. It starts at around 250MB/s, which isn't half bad write performance for an RAID 5 array of only 5 HDDS, but the copy I just did, consisting of 4 files totalling 12 GB, gradually declined to a 35MB/s low.
Now I guess one's advice would depend on a lot of metainfo, so here goes:
- The LSI card does not have a BBU (yet) so write-back is disabled.
- The HDDs are WD15EARS 2TB drives. Obviously these aren't the fastest HDDs out there, but a consistent 200MB/s isn't too much to ask I think.
- The SSDs are OCZ Vertex 2 60GB drives.
- Don't think it's relevant, but the HDDs have the spin down time upped to 5 minutes instead of the normal 8 seconds
- Drives show healthy in Storage Manager, no errors of note in logs
- Like I said, the SDDs are really fast, sporting up to 1100MB/s read speed, so that doesn't seem to be the bottleneck.
- Copy seems to pause, it'll run fast, stop, run fast again for about 500MB, etc etc, resulting in a lower speed overall.
- When creating the HDD array, I used a strip size of 512Kb. That's huge, but I'm expecting only large to huge files on that array. I'd rather not change that now either, as it would destroy existing data and I don't have a backup (yet)
- Operating system is Ubuntu 10.04 (64bit)
- Motherboard Asus WS Revolution (it's a workstation), 24GB of ECC RAM, Xeon W3570 at stock 3.2GHz
- The LSI card is inserted in the first PCIe slot (to avoid latency introduced by NF200)
- System is otherwise perfectly stable
- HDD array was formatted using "mkfs.ext4 -b 4096 -E stride=128,stripe-width=384 -L "DATA" /dev/sdb"
- fstab does not include data=writeback, nor noaccess, though I don't think that should be an issue influencing large files
Any and all advice is appreciated.