Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am planning to upgrade a free software mirror server, and I'd appreciate advice on setting up the new primary disks.

My current usage:

  • system software, config and workspace <30GB - stays on RAID-1 array
  • stuff that's on a single 900GB partition (four-disk RAID-5)
    • 2-3 small mirrors <1G
    • one 600MB mirror with lots of small files (portage)
    • two ~5GB mirrors
    • five 20-40GB mirrors
    • two 100GB mirrors
    • three 150GB mirrors

I will probably be getting 44TB of disks, which I would spread over three hardware RAID-5 arrays for a total of 34TB (plus spares).

I thought I'd make the arrays into LVM2 PVs and build a 34TB VG, which I'd somehow split up, making an LV for each mirror. Then I'd have an extN or XFS volume for each of the distributions.

One problem is that I can't really predict the growth of any of the mirrors. I may have to either create a lot of overhead in each LV or grow the LVs frequently. Major mirror shrinkage isn't really a concern; they keep getting bigger and bigger. Is there any real performance loss due to increased fragmentation if one resizes an LV many times?

I may want to optimize some of the file systems for specific workloads, such as small text files or CD images, so that's a strike against using a single FS. The multi-FS approach would let me track by-distro disk usage patterns more easily. A final possible disadvantage of keeping the one-big-FS is latency as the OS searches down the tree. How much of a concern is that?

I'll have 24 or 48GB of RAM, and I plan to be serving 30-50TB per month, with several big files (installers, CD images) hitting the cache and many 2-20MB files missing.

share|improve this question
up vote 1 down vote accepted

First of all, stay away from RAID. It is not worth it. With a 14TB array rebuilding is going to take days. You do not want to have your disks churning for days, then it is better to lose a part of the mirror and fetch the data again when the disk is replaced.

LVM is nice, and certainly something to use for your smaller mirrors, but I am not sure if it is so much help for main storage. The problem with LVM is that a failure of anything takes down the whole PV, so you don't want enormous PV's.

You might run into IO load balancing problems, which will force you to balance IO between the disks (for example, your Ubuntu mirror will probably be hit hard). Therefor I would recommend that you use some kind of layer that allows you to redistribute the IO load between the disks.

A typical solution when dealing with large and growing mirror storage is to create an abstraction layer that keeps track of where the files are located on disk (typically using a database), and then spreading out the files over multiple physical disks without redundancy. This is built into many NAS solutions.

You can find some more information in a previous answer here.

share|improve this answer

Considering the array size and the fact that you are going to expand it soon, I would recommend using RAID-6 instead of RAID-5 + spares for the array. On my hardware rebuild of 20TB array takes about 2-3 weeks, so if you use RAID-5 and a drive fails you will be at risk for extended period of time during rebuild. It is also well documented fact that many failures occur during rebuild which will be fatal for RAID-5 array.

I cannot comment on partitioning though. I personally avoid having too many partitions and I would rather have one big partition for everything (well maybe two :-), sometimes trading off potential performance gains in favor of manageability and convenience.

I was also practicing a small SSD for system partition lately, mostly because of reliability of SSD drives. This practice is still considered questionable by many though.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.