My disk are 10x1TB SAS 7200 RPM in a RAID 10 with a MegaRaid 9260 hardware controller with cache/BBU. This results in a 4.6TB RAID 10 volume.
hdparm -t (when device is empty) results in 500MB/s.
RAID chunk size is 64KB, filesystem block size is 2KB (I'm going to change it to the minimum chunk size and 4KB block size).
The directory pattern is /data/x/yz/zyxabc.gz
I'm using EXT4 with plans to move to XFS. The OS is RHEL 6.
As of now, it works great. The workload is 99% reads and it can read up to 300 files/second under normal conditions. The problem is backups. It takes 6 days to backup with scp. rsync is even slower. DD goes at about 2MB/s. LVM snapshots could be an option if I take the snapshot, back it up, and then delete it. Data consistency is very important to me.
Files are about 0.5-4KB each. Would I see increased backup performance if I stored all of the files in a database instead? What other alternatives are there for me to tackle the problem of backing up this many small files in a reasonable window?