Even from my limited experience I should say that your problem is under-specified. For example, you don't talk about load, latency, and bandwidth requirements. Check this Q&A page for more: Can you help me with my capacity planning?
My experience is building and managing 140TB (and growing, 80TB primary + 60 backup) storage for our lab. We use it as a storage for research data (imaging data mostly, some genomics; no databases). System consists of two servers in different server rooms, and was designed to be expandable. Please take what I write with a grain of salt.
Our servers run FreeBSD and ZFS, 4U boxes with SAS expanders. Each box is stuffed with 4-5TB drives. Backup is newer and has 12x5TB drives, configured as RAID6 aka RAIDZ2.
Expandability is achieved via two routes:
- Each Z Pool (e.g. 12x5TB) can be expanded by adding extra 12 HDDs in parallel. Trick is that we'll have to add drives to storage in chunks of 60TB, we can't (AFAIK) add 30TB and six months later extra 30TB. If you are interested in ZFS yo ushould read up on terminology (many websites, including Oracle's and FreeBSD's)
- Using SAS allows us easily add extra 4U JBOD box with up to 45 drive slots, by connecting to main box's HBA/RAID card via single cable.
So, from what I know:
- ZFS, when run on server-grade hardware, can be successfully used with almost no supervision for hosting 100TB of data and automatic off-site backups
- Expansion of ZFS storage can be easily done when using SAS expanders, if bandwidth requirements are met
- Expand-ability is not "smooth" in a sense that if you configure your zpool in blocks of 20TB you will have to add drives in chunks of 20TB or larger down the road. You can't just buy a 1TB hard drive and plug it in if you run some sort of RAID6/7. In case of RAID10 you can do that because you "block" is essentially 1 or 2 drives
Current system bandwidth limitations are as follows:
- Network via copper Ethernet 1Gbps (whole campus) or 10Gbps (inside
- RAID60 arrays using HDDs: n x 160MB/s (in 4x configuration
is it ~5Gbps)
- SAS3 protocol using 4x12=48 Gbps (connects backplane
to HBA using SAS3 4x cables)
As you can see, we are currently limited by 1Gbps ethernet, if we work inside the rack (storage<->analysis server), we become limited by server's ability to swallow 5Gbps streams. Our data is mainly large files (5-500GB), so I am not sure about the IOPS. To improve bandwidth currently we need to install fiber optics in the building (our lab and server rooms are on different floors), then we become limited by the speed of our RAID60 arrays. Then we either have to add extra stripes or build SSD-based storage. This system was built to work instead of USB3-based HDD, which it beats.