Which filesystem for 16TB under CentOS 5.5?

Question

We have a PowerVault NX3100 from Dell here. I have to install a Linux Server on this system and don't know which filesystem I should use and which partition layout would be useable for the RAID with 2x8 2TB disks (effectively 14TB, RAID 5).

I know that ext3 makes problems with fss > 2TB when used with default blocksize.

I tend to use XFS with two partitions of 8TB. Would you see this as a good choice? Is it possible (or recommandable) to use LVM for this system? Should I use only one or several fss? I would like to minimize the times for fsck or check_xfs.

Answer 1

Definitely XFS. XFS initialisation is much faster, performance is excellent, and XFS has been use for multi-terabytes volumes for ages. I currently support 230 machines with 8 to 76 TB XFS volumes. Tens are built with two or more RAID volumes aggregated through LVM without problem, so this is safe enough.

xfs_check speed depends mostly on the number of files. For typical large volumes (30 TB), xfs_repair takes less than 15 minutes given that the system has enough memory (older xfs_repair tends to gobbles tons of RAM), like 8 GB or more.

Answer 2

We have backup servers including 8-12TB of space, first one (10TB) running ext3 without any problems, and others running ext4.

ext4 seems to be much faster with large partitions and huge number of files. We use only single very large partition for data, as it is so much more flexible. And of course separate partition for root filesystem. We are using hardware RAID5 and LVM.

As of minimizing time for fsck, first advice would be shutting down server in right way whenever possible.

If you can easily split your filesystem in reasonable way to multiple smaller partitions, I would go with LVM and your filesystem of choice (ext3 is mature, and so is XFS. Many people are using ext4 without any problems). If you have smaller partitions, fsck will be faster, assuming that you don't have to fsck all partitions every time.

Of course you can also run fsck only on mandatory partitions at first (to get most important things up and running) and then continue running fsck on other partitions before mounting those.

Answer 3

Answer depends on many factors.

1) Supportability. Who is going to support the system? If you have a contract for the system support from an outside entity, then you have to stuck to the configuration they support. They may put some restrictions on what filesystems you can use.

2) Layout (number and type of filesystems) will depend on what you need. If two partitions of 8TB work for you, it may very well be the right choice. I assume we are talking about two partitions for data, and you have separate filesystems for the OS itself (/, /boot, /usr/, /tmp, /var).

3) File size and data access pattern. Removing large files on ext3 with default options (data=ordered) is painful. If you expect to need to remove large (hunderds of GBs) files often then either go to data=writeback with ext3, or choose xfs/ext4. The same problem will occur if you will have writers that sync significant amounts of data while there's other I/O going to the FS.

3) fsck time

o) I do not have experience with xfs fsck time and requirements.

o) ext3 fsck for 2x8TB will be problematic, because of memory requirmements. You will need RAM (certainly more than 2 GB) for it to complete, preferably a 64-bit system and a lot of time (I'd say a couple of days before fsck completes).

o) ext4 is supported on RHEL 5.6, and therefore should be on CentOS. This would reduce your fsck time dramatically compared to ext3 and should improve performance too. Personally I'd go with this filesystem, but that's because of

4) Comfort zone. From the filesystems that do the job choose one that you are most comfortable with. If you have years of experience with filesystem Z and it works for you, then probably this is a good filesystem for you.

My personal choice? separate /boot partition formatted with ext3. Rest of the storage made into LVM physical volumes. LVM with two volume groups -- one for the OS, one for the data. Leave some space unused in the volume groups, so that it is possible to do an LVM snapshot of the filesystem to fsck it without bringing the whole system down (with this big filesystems silent bit rot is possible, check for it, while waiting for btrfs to mature). ext4 for OS and data partitions -- because it is fast, I like it and trust it.