I am planning to upgrade a free software mirror server, and I'd appreciate advice on setting up the new primary disks.
My current usage:
- system software, config and workspace <30GB - stays on RAID-1 array
- stuff that's on a single 900GB partition (four-disk RAID-5)
- 2-3 small mirrors <1G
- one 600MB mirror with lots of small files (portage)
- two ~5GB mirrors
- five 20-40GB mirrors
- two 100GB mirrors
- three 150GB mirrors
I will probably be getting 44TB of disks, which I would spread over three hardware RAID-5 arrays for a total of 34TB (plus spares).
I thought I'd make the arrays into LVM2 PVs and build a 34TB VG, which I'd somehow split up, making an LV for each mirror. Then I'd have an extN or XFS volume for each of the distributions.
One problem is that I can't really predict the growth of any of the mirrors. I may have to either create a lot of overhead in each LV or grow the LVs frequently. Major mirror shrinkage isn't really a concern; they keep getting bigger and bigger. Is there any real performance loss due to increased fragmentation if one resizes an LV many times?
I may want to optimize some of the file systems for specific workloads, such as small text files or CD images, so that's a strike against using a single FS. The multi-FS approach would let me track by-distro disk usage patterns more easily. A final possible disadvantage of keeping the one-big-FS is latency as the OS searches down the tree. How much of a concern is that?
I'll have 24 or 48GB of RAM, and I plan to be serving 30-50TB per month, with several big files (installers, CD images) hitting the cache and many 2-20MB files missing.