RAID 5 reshape freeze

Question

I recently added a new disk to my RAID5 array and started growing it. Absent minded as I am I rebooted the server during this reshape process I as another program was hung and blocking some ports. Thinking of it now it might have been because the array hung but I cannot be sure.

I started the grow process with the following command:

$ mdadm --grow --raid-devices=4 /dev/md0

After this reboot the reshape process has frozen at 28%. I can no longer mount the array, stop it or anything it just seem to have frozen up.

Here is some info on the array:

# mdadm -D /dev/md0

/dev/md0:
        Version : 1.2
  Creation Time : Sat Mar 28 17:31:15 2015
     Raid Level : raid5
     Array Size : 5860063744 (5588.59 GiB 6000.71 GB)
  Used Dev Size : 2930031872 (2794.30 GiB 3000.35 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Sun Jun  7 11:04:28 2015
          State : clean, reshaping 
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 256K

 Reshape Status : 28% complete
  Delta Devices : 1, (3->4)

           Name : ocular:0  (local to host ocular)
           UUID : e1f7a83b:2e43c552:84d09d04:b1416cb2
         Events : 344582

    Number   Major   Minor   RaidDevice State
       4       8       17        0      active sync   /dev/sdb1
       1       8       49        1      active sync   /dev/sdd1
       3       8       65        2      active sync   /dev/sde1
       5       8       33        3      active sync   /dev/sdc1 

and

# cat /proc/mdstat 

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[4] sdc1[5] sde1[3] sdd1[1]
      5860063744 blocks super 1.2 level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
      [=====>...............]  reshape = 28.6% (840259584/2930031872) finish=524064.9min speed=66K/sec
      bitmap: 3/22 pages [12KB], 65536KB chunk

unused devices: <none>

Trying to mount the array just hangs

# mount /dev/md0 /mnt/storage/

And the same if I try to stop the array

# mdadm -S /dev/md0

I have also tried growing it down to 3 devices again but it is busy with the last reshape:

# mdadm --grow /dev/md0 --raid-devices=3

mdadm: /dev/md0 is performing resync/recovery and cannot be reshaped

I tried to mark the new drive as faulty to see if the reshape would stop but to no avail. It works to mark it as failed but nothing happens.

I also tried to run a check instead of a reshape (as I read somewhere this fixed a similar problem) but the device is busy

# echo check>/sys/block/md0/md/sync_action

-bash: echo: write error: Device or resource busy

What does this mean? I'm on really thin ice here with no idea what to do so any help is greatly appreciated.

EDIT

Pretty sure the reboot was not the cause of the problem. It seems to be some problem with the reshape that causes the array to hang. I get these errors in dmesg:

[  360.625322] INFO: task md0_reshape:126 blocked for more than 120 seconds.
[  360.625351]       Not tainted 4.0.4-2-ARCH #1
[  360.625367] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  360.625394] md0_reshape     D ffff88040af57a58     0   126      2 0x00000000
[  360.625397]  ffff88040af57a58 ffff88040cf58000 ffff8800da535b20 00000001642a9888
[  360.625399]  ffff88040af57fd8 ffff8800da429000 ffff8800da429008 ffff8800da429208
[  360.625401]  0000000096400e00 ffff88040af57a78 ffffffff81576707 ffff8800da429000
[  360.625403] Call Trace:
[  360.625410]  [<ffffffff81576707>] schedule+0x37/0x90
[  360.625428]  [<ffffffffa0120de9>] get_active_stripe+0x5c9/0x760 [raid456]
[  360.625432]  [<ffffffff810b6c70>] ? wake_atomic_t_function+0x60/0x60
[  360.625436]  [<ffffffffa01246e0>] reshape_request+0x5b0/0x980 [raid456]
[  360.625439]  [<ffffffff81579053>] ? schedule_timeout+0x123/0x250
[  360.625443]  [<ffffffffa011743f>] sync_request+0x28f/0x400 [raid456]
[  360.625449]  [<ffffffffa00da486>] ? is_mddev_idle+0x136/0x170 [md_mod]
[  360.625454]  [<ffffffffa00de4ba>] md_do_sync+0x8ba/0xe70 [md_mod]
[  360.625457]  [<ffffffff81576002>] ? __schedule+0x362/0xa30
[  360.625462]  [<ffffffffa00d9e54>] md_thread+0x144/0x150 [md_mod]
[  360.625464]  [<ffffffff810b6c70>] ? wake_atomic_t_function+0x60/0x60
[  360.625468]  [<ffffffffa00d9d10>] ? md_start_sync+0xf0/0xf0 [md_mod]
[  360.625471]  [<ffffffff81093418>] kthread+0xd8/0xf0
[  360.625473]  [<ffffffff81093340>] ? kthread_worker_fn+0x170/0x170
[  360.625476]  [<ffffffff8157a398>] ret_from_fork+0x58/0x90
[  360.625478]  [<ffffffff81093340>] ? kthread_worker_fn+0x170/0x170

Also, looking at CPU usage md0_raid5 seems to be having problems:

 PID USER      PR  NI    VIRT    RES  %CPU %MEM     TIME+ S COMMAND
 125 root      20   0    0.0m   0.0m 100.0  0.0  35:57.44 R  `- md0_raid5
 126 root      20   0    0.0m   0.0m   0.0  0.0   0:00.06 D  `- md0_reshape

Could this be why the reshape has stopped?

Is it possible to revert to using 3 drives again without losing data?

Answer 1

From man mdadm(8):

Changing the number of active devices in a RAID5 or RAID6 is much more effort. Every block in the array will need to be read and written back to a new location. From 2.6.17, the Linux Kernel is able to increase the number of devices in a RAID5 safely, including restarting an interrupted "reshape". From 2.6.31, the Linux Kernel is able to increase or decrease the number of devices in a RAID5 or RAID6.

So mdadm is perfectly capable of resume an interrupted grown. However, you must use a backup file (with --backup-file) in order to have resume (or abort) capability.

See also here for more information.

What was the exact command you run? Anyway, I suggest you to post your problem on the linux-raid mailing list.