Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I work in an scientific lab with a limited budget and ever-growing need for storage. A year ago we needed ~2-3 TB storage, today we need 13+ TB which fills our current server (linux, raid 6 with 9 drives), and it will only keep growing. The files are huge - 50GB+ each.

I want to build a server that:

a) Can handle uneven disk sizes, so we can replace older drives with larger ones as they become available in the market. b) Can handle additions of disks (that are potentially larger than any of other disk) after creation of the initial "volume". Ideally, I just want to pop in a drive into a hotswap bay and make it part of the volume. b) Has redundancy and can handle multiple disk failures. d) Quick 'fscking'. Last time our current server did this it took forever to get back up.

Can I do (a) and (b) with RAID? I know I can partition equal sized raid-partitions from unevenly sized disks, but I do not want to get into the business of micromanaging partitions with multiple raid arrays. Is ZFS an option? (FreeBSD would be acceptable too.)

Performance, honestly, is not much of an issue. It will only store and serve static content to a handful of researchers. Our LAN is 1Gbit and our WAN is only 100Mbit.

Any and all suggestions welcome.

share|improve this question
This doesn't address your main question, but as far as fsck speed: ext4 is much quicker than ext3. – Wyzard Jul 13 '12 at 2:46
Just to clarify, none of the processing takes place on the file server and thus none of the intermediate files end up on the server either... the raw data files (high throughput biological sequences) and the processed data is all that resides there. Even then we are at 13 TB. – user1481 Jul 13 '12 at 6:05
up vote 6 down vote accepted

I would recommend ZFS-based solutions, but running a purpose-built OS (NexentaStor), versus trying to run additional applications on the system. This gives you the flexibility to treat your storage as an appliance and eliminates application dependencies. Export to your Linux systems via NFS or iSCSI.

The remainder of your requirements are well-covered by ZFS solutions. Do you have a budget established?

I recommend going with an integrator/partner who can help design a robust system and alleviate any scaling/longevity issues. Chances are that they've seen a situation like yours or have dealt with similar requirements. However, if you go it on your own, do your due diligence and avoid the mistakes others have made.

Good places to start:

share|improve this answer

Divorce your storage needs from your processing needs. You can purchase a san for relatively low prices (HP P2000's are cheap if you're not buying them fully populated, and you can expand them up to 6 shelves of 3TB disks, for 36x6 TB of storage RAW), and if you stick with iSCSI you can avoid costly HBAs and fibre. iSCSI would also let you continue utilizing the server you're currently using as your file server, it just lets you attach a bunch of disks. You can also look at older models like MSA 2000's. Dell also has some offerings which are decent. Then, connect it to your server with a few initial disks (2 for raid 1, 3 for raid 5 or raid 1e, 4 for raid 1+0 or raid 6 depending on your storage requirements).

On the server, use LVM to make a physical volume out of these, and create a volume group/logical volume on top of that. In the future you can add additional disks to the SAN chassis, and add them as new physical volumes, and then expand the logical volume. Then when you max out capacity of your SAN's disk shelf, you just add another shelf with a couple disks, and grow the same way as the first shelf. We use this for many of our storage systems. Some careful planning can help you keep from having data for the same logical volume hosted on two different shelves (in case some idjit decides they want to trip over a cable connecting the two san shelves).

For the filesystem, ext3, ZFS, ext4 all support increasing the size of the filesystem on the fly. However if you are ever concerned you'd need to shrink it, lean towards ext3. It's better documented and supported. This solution is simple, doesn't require a consultant, and allows you to grow your solution with your dataset.

You really want to remove Storage from your server though if you need to expand it so much.

share|improve this answer

I think ewwhite is right here, for something this large, a dedicated, commercially-supported appliance is what I'd go with.

Otherwise, you could slap something together with Linux using md to add pairs of mirrored drives (the drives in the pair would have to be the same size as each other but could be different sizes than the old drives), then using LVM to incorporate that new pair into your filesystem either by migrating logical volumes off of old physical volumes and/or growing them onto the new physical volume. Redundancy would be an issue (you could lose half of the drives, but only if they're the right half), but you'd have a lot more flexibility on adding new drives and removing old drives.

... just be sure you have backups.

share|improve this answer

One thing to consider is that ZFS does not support removing vdevs, at all. That means that if you ill-advisedly expand your array, or you decide you want to switch from, eg, RAID6 to RAID10, you're out of luck. As long as your disk layout is fixed and you're upgrading every disk in a vdev (eg, both disks in a mirror), I agree ZFS is probably a good solution.

BTRFS does support device removal, for their equivalent of RAID10. It has gotten a lot more stable in the last couple of years as well.

If you're willing to consider Windows, Windows Storage Spaces is actually a pretty good solution to needing to throw a bunch of heterogeneous disks in a server with redundancy and have them play well.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.