21

I have a Linux server with many 2 TB disks, all currently in a LVM resulting in about 10 TB of space. I use all this space on an ext4 partition, and currently have about 8,8 TB of data.

Problem is, I often get errors on my disks, and even if I replace (that is to say, I copy the old disk to a new one with dd then i put the new one in the server) them as soon as errors appear, I often get about 100 MB of corrupted data on it. That makes e2fsck go crazy everytime, and it often takes a week to get the ext4 filesystem in a sane state again.

So the question is : What would you recommend me to use as a filesystem on my LVM ? Or what would you recommend me to do instead (I don't really need the LVM) ?

Profile of my filesystem :

  • many folder of different total sizes (some totalling 2 TB, some totalling 100 MB)
  • almost 200,000 files with different sizes (3/4 of them about 10 MB, 1/4 between 100 MB and 4 GB; I can't currently get more statistics on files as my ext4 partition is completely wrecked up for some days)
  • many reads but few writes
  • and I need fault tolerance (I stopped using mdadm RAID because it doesn't like having ONE error on the whole disk, and I sometimes have failing disks, that I replace as soon as I can, but that means I can get corrupted data on my filesystem)

The major problem are failing disks; I can lose some files, but I can't afford lose everything at the same time.

If I continue to use ext4, I heard that I should best try to make smaller filesystems and "merge" them somehow, but I don't know how.

I heard btrfs would be nice, but I can't find any clue as to how it manages losing a part of a disk (or a whole disk), when data is NOT replicated (mkfs.btrfs -d single ?).

Any advice on the question will be welcome, thanks in advance !

6
  • 1
    Exactly what disk errors you get. That should give a clue Oct 8, 2012 at 18:00
  • Bad sectors, often it's only one or two bad sectors on the whole disk ...
    – alphatiger
    Oct 8, 2012 at 18:14
  • That means your disk is going bad. Hardly anything to do with filesystem. If the disk is bad, no matter what fs you use, will be handy. As others have mentioned, go for RAID disks and/or buy enterprise disks. Also, look for quality controllers too. Oct 8, 2012 at 18:27
  • Yep, I know, that's why I replace disks that are going bad. Sorry if my question wasn't clear. But still, I thought that some filesystems would behave better with corrupted data ...
    – alphatiger
    Oct 8, 2012 at 18:36
  • You really really should replace the faulty pieces of your hardware. This is like looking at a crash test dummy after a car has been driven against the wall 200 km/h. "Oh look! His left leg is almost OK! The test was successful!" ... no filesystem can help you if the underlying hardware rots. XFS has faster fsck than ext*, and after enough time passes and the filesystem matures a bit more, perhaps btrfs would work, too. Then there's ZFS but on Linux its state is a bit sad. Oct 8, 2012 at 18:45

7 Answers 7

22

It's not file system problem, it's disks' physical limitations. Here's some data:

SATA drives are commonly specified with an unrecoverable read error rate (URE) of 10^14. That means that 1 byte per 12TB will be unrecoverably lost even if disks work fine.

This means that with no RAID you will lose data even if no drive fails - RAID is your only option.

If you choose RAID5 (total capacity n-1, where n = number of disks) it's still not enough. With 10TB RAID5 consisting of 6 x 2TB HDD you will have a 20% chance of one drive failure per year and with a single disk failing, due to URE you'll have 50% chance of successfully rebuilding RAID5 and recovering 100% of your data.

Basically with the high capacity of disks and relatively high URE you need RAID6 to be secure even again single disk failure.

Read this: http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162

7
  • 3
    Wait, URE means Unrecoverable Read Error but this doesn't mean that the disk actually HAS the error. The next read may (and probably will) return the correct bit. The OS will probably just re-read the sector and obtain the correct data. You also forgot to talk about S.M.A.R.T.: before a sector is permanently damaged, S.M.A.R.T. will try to read/write data from/to it. If it detects too many failures, S.M.A.R.T. simply moves the content of the sector in another place and marks the sector as BAD and nobody will be able to write onto it again.
    – Avio
    Oct 9, 2012 at 13:17
  • So, you are simply suggesting to buy tons of disks without asking WHY his disks are so faulty. It could be a heat problem, it could be a problem with a faulty SATA controller, it could be a problem of bad SATA connectors, etc. etc. etc.
    – Avio
    Oct 9, 2012 at 13:20
  • @Avio What I'm saying is that with 10TB of data you will have read errors due to hard disk limitations, even if all disks, SATA controller, SATA connectors etc are in perfect condition and working according to specs. I am also saying that even if you decide to use RAID to mitigate that you should go with RAID6 because disk capacity + URE make even RAID5 not reliable enough. Even single drive failure on RAID5 has a high (50% FFS!) data loss chance.
    – c2h5oh
    Oct 9, 2012 at 13:28
  • 1
    @Avio U in URE stands for Unrecoverable as in gone for good.
    – c2h5oh
    Oct 9, 2012 at 13:37
  • It can be the file systems problem, if you use a copy on write filesystem like btrfs or xfs you can very likely recover a previous version of the file, so only loosing the last change to the file. (if it was ever changed) Oct 12, 2012 at 14:21
13

Do yourself a favor and use a RAID for your disks, could even be software RAID with mdadm. Also think about why you "often get errors on your disks" - this is not normal except when you use cheap desktop class SATA drives instead of RAID grade disks.

After that, the filesystem is not that important anymore - ext4, xfs are both fine choices.

4
  • 1
    I agree that I should ;) but I don't use RAID for many reasons. Main one is the price, as they are 2-3 times more expensive, and I can't really afford that. The second reason is that last time I used RAID 5, I was lucky enough to get two bad disks before I could connect a new one and resync it (I didn't have any spare disks at the time, I had to wait for a new one; I agree that with RAID class disks, I would have had this problem). The third reason is that as the data I have to store is growing, I add new disks of greater sizes progressively, what I cannot do with a RAID configuration.
    – alphatiger
    Oct 8, 2012 at 18:28
  • So I'm trying to see if there exists a filesystem that someone would recommend me to use in a configuration where I can't rely on uncorrupted data. Still, thanks for your answer !
    – alphatiger
    Oct 8, 2012 at 18:32
  • 4
    So you are saying your data is not worth the additional expense? If you can't afford to have at least two copies of your data, then you should consider it lost. You are right that RAID5 probably isn't a good choice, you should probably looking at RAID6 or RAID10.
    – Zoredache
    Oct 8, 2012 at 19:52
  • @alphatiger: The discs are only too expensive if your time and your data are too cheap. Oct 11, 2012 at 13:20
8

I've had good luck with ZFS, you could check to see if it's available on whatever distro you use. Fair warning, it'll probably mean rebuilding your whole system, but it gives really good performance and fault-tolerance.

4
  • I currently use Debian GNU/Linux, it seems there's a FUSE implementation, but no package (due to licensing problems). I'll probably give it a try (after compiling from sources, as using FUSE it not very nice for high output), I don't worry about having to rebuild my whole filesystem. Thanks !
    – alphatiger
    Oct 8, 2012 at 18:46
  • +1 for ZFS. Traditional RAID will silently corrupt data because it's not intelligent enough to know when blocks are wrong, or how to repair them. ZFS on the other hand will detect corrupt blocks (via checksums) and repair them from known good mirror copies. Running ZFS under FUSE, while not ideal, will perform well enough for many workloads. That being said, you should load test your application before using this in a production environment.
    – bahamat
    Oct 8, 2012 at 23:47
  • 1
    Another +1 for ZFS. Pretty much all the servers here are running Linux and I'm a big fan of it, but ZFS has proven so useful to me in the last 3+ years that I've actually gone through the effort of learning and setting up FreeBSD on the big storage machine to be able to use ZFS without any licensing or performance issues.
    – ssc
    Oct 15, 2012 at 19:53
  • I'm running it under Solaris on my old Sun workstation, and the performance is nothing short of amazing, considering the hardware (single-core Opteron @ 2.2GHz with 3G of memory and a pair of 250G SATA drives).
    – TMN
    Oct 15, 2012 at 20:15
8

I add new disks of greater sizes progressively

Since you are interesting in using LVM, and you want to handle multiple drives, the simple answer would be to just use the mirror feature that is part of LVM. Simply add all the physical volumes into your LVM. When you are creating a logical volume pass the --mirrors option. This duplicates your data.

Another option might be to just setup several RAID1 pairs. Then add all the RAID1 volumes as PVs to your VG. Then whenever you want to expand your storage, just buy a pair of disks.

7

You should really be using a RAID 5, 6, 10, 50, or 60. Here's some resources to get you started:

background info about RAIDs

howto's & setup

Check out my delicious links for additional RAID links: http://delicious.com/slmingol/raid

2
  • See my comments on SvenW's answer to see why I don't really want RAID. (In fact, I already did setup multiple software RAIDs in a company who could afford that ...) Still, thanks !
    – alphatiger
    Oct 8, 2012 at 18:34
  • I've always used commodity drives for RAIDs, never used ones rated for RAID use and have never had issues with that so long as you pick a RAID that has enough redundancy within it (RAID 6 or RAID 60). Using a RAID 6 you need an even number. You can grow RAIDs fairly easily by replacing existing members with larger disks and then growing into the newer disks' space.
    – slm
    Oct 8, 2012 at 19:02
4

If you're really worried about data corruption, I would recommend a checksummed filesystem such as zfs and btrfs -- though note that btrfs is still considered to be in-development and not production-ready.

There is no gurantee that the data read (even successfully read) from a disk will be correct. Blocks have checksums, but they're simple checksums that don't always catch errors. Newer filesystems like ZFS attach more capable checksums to files and can (and reportedly do) catch and repair data errors not noticed by the hard disk or RAID controller.

1

As @c2h5oh says, the Unrecoverable is critical - it means the disk has already tried and failed to re-read the sector.

In my experience, once a disk starts producing unrecoverable read errors (UREs), some data is lost forever, and your only hope is to immediately backup all data using GNU ddrescue, which can retry the failing sectors as well as skip unrecoverable ones.

Assuming you have backups, they may well have failed due to the UREs, and will certainly have some corrupt files, so you will have to piece together a full set of data from various backups of the same filesystem.

The other answers recommending ZFS are worth reading, as its continuous data scrubbing and RAID features will help keep your data safer in future - though still not a substitute for backups, which also protect against user and admin errors.

I would only use LVM if you don't need snapshots - it doesn't integrate so well with RAID, doesn't include data scrubbing / data checksums, and you still need backups, so something like ZFS is probably a better option. See this answer on LVM problems and risks for more.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .