Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm getting a rather odd result from df from one of my disks using mdadm. It's setup as 4 of 2TB disks in raid 10.

# df
Filesystem            Size  Used Avail Use% Mounted on
/dev/md2              3.6T   40G  3.4T   2% /  <------ this one
tmpfs                 7.8G     0  7.8G   0% /dev/shm
/dev/md0              4.9G  189M  4.4G   5% /boot

The actual disk usage is roughly around 2TB. Yet, it's reporting only 40GB of usage.

I see some errors like this in /var/log/messages as well as /var/log/dmesg (same ones)

Feb 13 05:46:00 las kernel: EXT4-fs error (device md2): ext4_mb_generate_buddy: EXT4-fs: group 24449: 0 blocks in bitmap, 32768 in gd
Feb 13 05:46:00 las kernel: EXT4-fs error (device md2): ext4_mb_generate_buddy: EXT4-fs: group 24450: 3 blocks in bitmap, 32771 in gd
Feb 13 05:46:00 las kernel: EXT4-fs error (device md2): ext4_mb_generate_buddy: EXT4-fs: group 24451: 6 blocks in bitmap, 32766 in gd
Feb 13 05:46:00 las kernel: EXT4-fs error (device md2): ext4_mb_generate_buddy: EXT4-fs: group 24452: 50 blocks in bitmap, 32742 in gd
Feb 13 05:46:00 las kernel: EXT4-fs error (device md2): ext4_mb_generate_buddy: EXT4-fs: group 24453: 43 blocks in bitmap, 32768 in gd
Feb 13 05:46:00 las kernel: EXT4-fs error (device md2): ext4_mb_generate_buddy: EXT4-fs: group 24454: 30 blocks in bitmap, 32768 in gd
Feb 13 05:46:00 las kernel: EXT4-fs error (device md2): ext4_mb_generate_buddy: EXT4-fs: group 24455: 77 blocks in bitmap, 32768 in gd
Feb 13 05:46:00 las kernel: EXT4-fs error (device md2): ext4_mb_generate_buddy: EXT4-fs: group 24456: 27 blocks in bitmap, 32744 in gd
Feb 13 05:46:00 las kernel: EXT4-fs error (device md2): ext4_mb_generate_buddy: EXT4-fs: group 24457: 68 blocks in bitmap, 32265 in gd
Feb 13 05:46:00 las kernel: EXT4-fs error (device md2): ext4_mb_generate_buddy: EXT4-fs: group 24458: 32 blocks in bitmap, 1804 in gd
Feb 13 05:46:00 las kernel: JBD: Spotted dirty metadata buffer (dev = md2, blocknr = 0). There's a risk of filesystem corruption in case of system crash.
Feb 13 05:46:00 las kernel: JBD: Spotted dirty metadata buffer (dev = md2, blocknr = 0). There's a risk of filesystem corruption in case of system crash.
Feb 13 05:46:00 las kernel: JBD: Spotted dirty metadata buffer (dev = md2, blocknr = 0). There's a risk of filesystem corruption in case of system crash.

But unsure what to make of it.

mdadm detail shows...

# mdadm --detail /dev/md2
/dev/md2:
        Version : 1.1
  Creation Time : Thu Oct 18 22:20:38 2012
     Raid Level : raid10
     Array Size : 3896783872 (3716.26 GiB 3990.31 GB)
  Used Dev Size : 1948391936 (1858.13 GiB 1995.15 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Wed Feb 13 05:58:12 2013
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 512K

           Name : i2274.userdns.net:2
           UUID : f64e69c7:8342cdd1:0a275bbf:3ba052f4
         Events : 275873

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2
       2       8       33        2      active sync   /dev/sdc1
       3       8       49        3      active sync   /dev/sdd1

Lastly, I just ran fsck by echo y > /forcefsck and reboot, but seems nothing have changed. I think the disk is corrupted, but uncertain how to proceed.

share|improve this question
    
Does /forcefsck cause a FULL fsck? If not then you might be right things are still corrupt. –  mdpc Feb 13 '13 at 6:22
1  
/forcefsck forces a full fsck, it appends on a -f to the fsck command like, verified per /etc/rc.sysinit on centos 5 –  kormoc Feb 13 '13 at 6:40
    
You should boot off of a rescue disk and fsck by hand, interactively it might ask you if it should do some potentially unsafe fixes that it would bypass in non-interactive mode. Backups are a must. –  kormoc Feb 13 '13 at 6:41
    
What filesystem? Directly on md0 or on a partition on it? Any comments by S.M.A.R.T.? Any fallout in /var/log/messages or its neighbors? –  vonbrand Feb 17 '13 at 18:57
add comment

1 Answer

up vote 0 down vote accepted

Just to bring closure to this issue, I had it do a complete fsck, like @kormoc suggested, in non-interactive mode on boot. This has resolved the issue.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.