Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I recently added a 7th 2TB drive to a linux md software RAID 6 setup. After md finished reshaping the array from 6 to 7 drives (from 8 to 10TB), I was still able to mount the file system without problems. In preparation for resize2fs, I then unmounted the partition and ran fsck -Cfyv and was greeted with an endless stream of millions of random errors. Here is a short excerpt:

Pass 1: Checking inodes, blocks, and sizes
Inode 4193823 is too big.  Truncate? yes
Block #1 (748971705) causes symlink to be too big.  CLEARED.
Block #2 (1076864997) causes symlink to be too big.  CLEARED.
Block #3 (172764063) causes symlink to be too big.  CLEARED.
...
Inode 4271831 has a extra size (39949) which is invalid Fix? yes
Inode 4271831 is in use, but has dtime set.  Fix? yes
Inode 4271831 has imagic flag set.  Clear? yes
Inode 4271831 has a extra size (8723) which is invalid Fix? yes
Inode 4271831 has EXTENTS_FL flag set on filesystem without extents support. Clear? yes
...
Inode 4427371 has compression flag set on filesystem without compression support. Clear? yes
Inode 4427371 has a bad extended attribute block 1242363527.  Clear? yes
Inode 4427371 has INDEX_FL flag set but is not a directory. Clear HTree index? yes
Inode 4427371, i_size is 7582975773853056983, should be 0.  Fix? yes
...
Inode 4556567, i_blocks is 5120, should be 5184.  Fix? yes
Inode 4566900, i_blocks is 5160, should be 5200.  Fix? yes
...
Inode 5628285 has illegal block(s).  Clear? yes
Illegal block #0 (4216391480) in inode 5628285.  CLEARED.
Illegal block #1 (2738385218) in inode 5628285.  CLEARED.
Illegal block #2 (2576491528) in inode 5628285.  CLEARED.
...
Illegal indirect block (2281966716) in inode 5628285.  CLEARED.
Illegal double indirect block (2578476333) in inode 5628285.  CLEARED.
Illegal block #477119515 (3531691799) in inode 5628285.  CLEARED.

Compression? Extents? I've never had ext4 anywhere near this machine!

Now, the problem is that fsck keeps dying with the following error message:

Error storing directory block information (inode=5628285, block=0, num=316775570): Memory allocation failed

At first I was able to simply re-run fsck and it would die at a different inode, but now it's settled on 5628285 and I can't get it to go beyond that.

I've spent the last days trying to search for fixes to this and found the following 3 "solutions":

  • Use 64-bit linux. /proc/cpuinfo contains lm as one of the processor flags, getconf LONG_BIT returns 64 and uname -a has this to say: Linux <servername> 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux. Should be all good, no?
  • Add [scratch_files] / directory = /var/cache/e2fsck to /etc/e2fsck.conf. Did that and every time I re-run fsck, it adds another 500K *-dirinfo-* and an 8M *-icount-* file to the /var/cache/e2fsck directory. So that seems to have its desired effect as well.
  • Add more memory or swap space to the machine. 12GB of RAM and a 32GB swap partition should be sufficient, no?

Needless to say: Nothing helped, otherwise I wouldn't be writing here.

Naturally, now the drive is marked bad and I can't mount it any more. So, as of right now, I lost 8TB of data due to a disk-check?!?!?

This leaves me with 3 questions:

  • Is there anything I can do to fix this drive (remember, everything was fine before I ran fsck!) other than spending a month to learn the ext3 disk format and then trying to fix it manually with a hex editor???
  • How is it possible, that something as mission-critical as fsck for a file-system as popular as ext3 still has issues like this??? Especially since ext3 is over a decade old.
  • Is there an alternative to ext3 that doesn't have these sorts of fundamental reliability issues? Maybe jfs?

(I'm using e2fsck 1.42.5 on 64-bit Debian Wheezy 7.1 now, but had the same issues with an earlier version on 32-bit Debian Squeeze)

share|improve this question
5  
I don't want to rub it in, but no filesystem is reliable enough to make it possible to not have a backup, especially before such dangerous operations. So, just start your restore and if you don't have a backup, don't blame anyone but yourself. –  Sven Jun 21 '13 at 18:37
2  
fsck -y is a bad idea. You always want to know that there are problems before you decide to do something about it. In your case, you could have noticed that fsck was reporting way too many errors and investigated before actually making any changes. –  longneck Jun 21 '13 at 18:40
2  
I make this a comment as it would be downvoted so much otherwise: use a decent filesystem instead of extN filesystems. Use XFS if you want a simpler solution. Or use ZFS. For this kind of scenario ZFS just works better. –  cstamas Jun 21 '13 at 19:23
1  
@MarkusA. Again, no. Your analogy is completely off-base. It's more like air bags reduce crash fatalities by 80%, but they also cause an additional 1% of injuries that would not have occurred. Statistically you're better off with air bags even though you're trading one type of trouble (the 80% reduction) for another (1% of cases where the airbag causes an injury). Similarly, just because fsck seemingly made your problem worse doesn't mean the tool is flawed. Remember: by specifying -y you told fsck to blindly fix without warning. You chose to drive with your nose against the airbag. –  longneck Jun 21 '13 at 19:30
2  
I think you're being a little harsh on ext3, here. You performed two majorly dangerous things, the first being to try to expand the number of columns in a live RAID, the second being resizing a file system. The second of those is not a regular happening, and the code paths are not all that well trodden. But the first is crazy dangerous; I've never done it in 20+ years of UNIX sysadmin. If you don't punch ext3 in the face, it probably won't beat up on you; but if you do - and you did - then you can't be too surprised when this sort of thing happens. –  MadHatter Jun 21 '13 at 20:38

2 Answers 2

Just rebuild the array and restore the data from a backup. The whole point of RAID is to minimize downtime. By messing around and trying to fix a problem like this, you just increase your downtime defeating the whole purpose of RAID. RAID doesn't protect against data loss, it protects against downtime.

share|improve this answer
up vote 2 down vote accepted

After playing around with fsck some more, I found some remedies:

Preventing the 'Memory allocation failed' error

fsck seems to have a major issue with memory leakage. If it is run on a file-system with some problems (real or imaginary), it will "fix" them one-by-one (see screen dump in original question). As it does so, it consumes more and more memory (maybe keeping a change-log?). Pretty much without bounds. But, fsck can be cancelled at any time (Ctrl-C) and restarted. In this case, it will continue where it left off, but it's memory use is reset to next-to-nothing (for a while).

With this in mind, the three things that need to be done are:

  • Use 64-bit Linux (it seems to make a difference in how fsck can use the available memory)
  • Add a ridiculously huge swap partition (I used 256GB, fsck runs for about 12 hours with it)
  • Frequently abort and restart fsck (how frequently depends on the size of the swap partition)

NOTE: I have no idea if canceling and restarting fsck brings with it any other dangers (probably does), but it seems to work for me.

Dealing with the resulting damage, if the 'Memory allocation failed' error occurs (IMPORTANT!)

fsck handles the Memory allocation failed error in the worst possible way: I destroys perfectly good data. I'm not sure why, but my guess is that it does some final data-write to disk of things that it had kept in memory, which (due to the error) have meanwhile gotten corrupted.

In my case, the most visible problem was that when I restarted fsck after the error, it sometimes reported a corrupted super-block. The problem is: I have no idea how corrupted the super-block was, especially in the cases where it didn't report it as corrupted. Maybe, if restarted after the error, it then uses incorrect drive meta-data found in the corrupted super-block to do all further checks and ends up fixing "issues" that aren't really there, destroying good data in the process.

Therefore, if fsck ever dies with the Memory allocation failed error, it needs to be restarted using the -b parameter to use a backup super-block that (hopefully) wasn't corrupted by the error. The location of the backup super-blocks can be found using mke2fs -n /dev/....

Since I don't know what happens if fsck dies with the backup super-block selected, I usually just abort fsdk immediately when it gets to Pass 1: Checking inodes, blocks, and sizes and restart it again without -b, at which point it starts without complaining about a bad super-block. I.e. it seems like the first thing fsck -b does is to restore the main super-block.

Now the one we've all been waiting for:

How to mount a file-system without letting fsck run to completion

This, I found by accident: It turns out that after running fsck -b and aborting it as soon as it prints the Pass 1: Checking inodes, blocks, and sizes (before any errors are found) the file-system is left in a mountable state (Yay! I got pretty much all of my data back!).

(Note: There may be another way using mount -o force, but it wasn't needed in my case.)

How to avoid all these issues in the first place

There seem to be two ways:

  • Use ext3, but keep a perfectly up-to-date backup. Then, frequently run fsck with parameter -N. If it shows any problems, delete the entire fs and restore everything from the backup. Since, in this scenario, one would be relying very heavily on the backup, I suggest keeping a backup of the backup. Also, use a copy-tool that somehow ensures that the restore does not create random errors in the process (An MTBF of a trillion r/w-ops is small when dealing with TB's of data). Make sure to plan for the resulting down-time, too, as a multi-TB restore probably takes a while...
  • My recommendation: Do NOT use ext3! The fs-design and associated tools (here fsck) aren't robust enough for real production use (yet?). The way fsck handles the memory error and the fact that the error occurs in the first place are not acceptable in my mind. I will be trying xfs from now on, but don't yet have enough experience with it to tell whether it's any better.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.