Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm using mdadm for several RAID1 mirrors. md7 is an N-way mirror consisting of 3 spinning disks (all flagged write-mostly) and an SSD:

md7 : active raid1 sdd1[0] sde5[3](W) sdf5[4](W) sdc1[1](W)
      234428416 blocks [4/4] [UUUU]

md6 : active raid1 sdf6[0] sde6[1]
      1220988096 blocks [2/2] [UU]

md2 : active raid1 sdb6[0] sda6[1]
      282229824 blocks [2/2] [UU]

md1 : active raid1 sdb2[0] sda2[1]
      19534976 blocks [2/2] [UU]

md0 : active raid1 sdb1[0] sda1[1]
      192640 blocks [2/2] [UU]

The entire system has hung 3 times in the past 2 weeks, requiring a hard reset. For the time being, I'm going to assume the system hang is unrelated to my md issue, although I can't completely discount that possibility. Each time we've rebooted, md7 has required a rebuild, but I can't figure out how to tell from the logs which disk triggered the rebuild. I thought iostat might be able to help me while the RAID was still rebuilding:

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              43.39      1038.34       558.83     223108     120075
sdb              66.88      1445.47       648.86     310588     139420
sdc              36.42        12.99     22256.81       2792    4782320
sdd             190.75     23227.78       331.14    4990954      71152
md0               2.11        21.39         0.23       4596         50
md1             173.72      1855.87       522.14     398770     112192
md2              11.68        65.84        27.59      14146       5928
md6              27.42       149.83        69.51      32194      14936
sde              75.83        70.81     22326.91      15214    4797384
sdf              79.31        99.41     22326.91      21360    4797384
sr0               0.04         2.61         0.00        560          0
md7             202.31      1287.41       331.07     276626      71136

...but it looks to me like md7 is using sdd to rebuild all the other disks in that RAID. I thought maybe this was simply because sdd is an SSD and all the other disks are marked write-mostly, but in that case, it should only rebuild the one disk that was out of sync (unless all the spinning disks just happened to be out of sync, which seems unlikely to me).

Another theory that I have is that all the spinning disks are always out of sync upon reboot simply because the SSD's writes are so fast that it has time to finish writing a block while the others are still writing, then the system just happens to lock up before the other disks finish writing that block?

So, how do I tell which disk(s) triggered the resync? Is the fact that I have an n-way mirror with mixed SSD and spinning disks possibly responsible for the fact that all the spinning disks are always rebuilt after one of these freezes, or does the md driver guarantee that a block isn't considered written on one disk until it's successfully written on all disks?

share|improve this question
    
This happened to me last month, smartd or /var/log/messages should show something, no? –  luckytaxi Nov 15 '12 at 17:55
    
The machine froze up in all 3 instances and we had to do a hard reset, so there is no information logged. /var/log/messages shows that md7 was resynced upon the subsequent boot, but as far as I can tell, there is no indication which disk(s) needed to be resynced. –  rob Nov 15 '12 at 20:11
add comment

1 Answer

I understand that (at least linux) raid works something like a filesystem for these purposes - if the system crashes while it's in use, it will need to be checked on reboot. So the cause of your system's crashes may not be any disks in the array.

share|improve this answer
    
I already specified in the question that I am assuming for now that the system hang is unrelated (because a properly-behaving system should kick the bad disk off and e-mail me a DegradedArray event). What I want to do is find out (by querying a device or looking at a log) which devices are being resynced, and why. I can infer from iostat that all three spinning disks are being resynced, but I don't know how to confirm this. –  rob Nov 15 '12 at 20:23
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.