8

This afternoon, someone at our office decided to pull the plug out of our server because it was storming outside. They didn't shut it down, they just pulled the plug out while it was running.

The server has 4 SATA drives in a software RAID 10 configuration, and LVM running on top of the RAID. The Server is running CentOS 6.2 Minimal and is a virtual machine host using KVM. At the time that it was unplugged, there were many guest machines running on the computer. Each guest has one or more LVM partitions that it uses directly as hard drives. The guest partitions are EXT3, EXT4 and NTFS. The Host OS is on an EXT4 partition.

Later, when the power came back, that person plugged it back in, and it started up. Since they plugged it in without attaching a monitor first, there is no way to see what came up on the screen. I tried attaching a monitor now, but it won't work unless the monitor is connected at boot. I've left it on, exactly as is, until I can get some advice since I don't want to screw anything up (further).

I can get into the host via SSH. I have not rebooted it yet in case there is something in a log somewhere that might be useful.

What I need to do is check all the disks and partitions for data integrity, if that's even possible. I think RAID 10 uses some kind of memory based cache and I'm worried about the drives being inconsistent, or files being corrupt if there were things in the cue to write to the drive that hadn't been written yet.

[root@othello ~]# cat /proc/mdstat
Personalities : [raid10] [raid1] 
md2 : active raid1 sdc1[2] sda1[0] sdd1[3] sdb1[1]
      102388 blocks super 1.0 [4/4] [UUUU]

md0 : active raid10 sda3[0] sdc3[2] sdd3[3] sdb3[1]
      1952289792 blocks super 1.1 512K chunks 2 near-copies [4/4] [UUUU]
      bitmap: 0/15 pages [0KB], 65536KB chunk

md1 : active raid10 sdc2[2] sda2[0] sdd2[3] sdb2[1]
      1022976 blocks super 1.1 512K chunks 2 near-copies [4/4] [UUUU]

unused devices: <none>

It also bothers me that it's calling my arrays, "near-copies". Is that normal?

What kind of disk checks should I run to make sure everything is OK with the drives and data? Are there any other things I should check?

UPDATE

Output of mdadm --detail

[root@othello ~]# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.1
  Creation Time : Sat Feb 25 09:26:20 2012
     Raid Level : raid10
     Array Size : 1952289792 (1861.85 GiB 1999.14 GB)
  Used Dev Size : 976144896 (930.92 GiB 999.57 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Sun Mar 11 12:59:30 2012
          State : active 
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 512K

           Name : othello.myserver.com:0  (local to host othello.myserver.com)
           UUID : 58ba40ab:12516733:e3779362:68200fdd
         Events : 2208

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       8       19        1      active sync   /dev/sdb3
       2       8       35        2      active sync   /dev/sdc3
       3       8       51        3      active sync   /dev/sdd3
Improve this question

2 Answers 2

Reset to default
3

The RAID is fine, all UUUU's means all disks in the array are up. I'd not even worry about that for now.

As for the VM's, if you want to run fscks on them, stop the VM's and run

fsck.ext3 (ext4, etc) /path/to/lvm (usually like /dev/vg-name/lv-name)

If you are using KVM, you should be able to use virsh to do anything you need to the VM's. Here is a link to the virsh man page http://linux.die.net/man/1/virsh

If you really want to run disk checks on your raid arrays, you'll have to reboot into single user mode or boot from a live cd so you can fsck the individual /dev/mdX devices. Since the primary filesystem is EXT4, I'd not bother, it's much better than EXT3 with power outages.

Improve this answer
1
  • +1, will try tomorrow.
    – Nick
    Mar 14, 2012 at 23:40
1

Try mdadm --detail /dev/md0 (same for md1 and md2).

Then try the advice given here: http://linas.org/linux/raid.html

Improve this answer
3
  • I've posted the output of mdadm --detail /dev/md0 above. I read through the guide you linked, but it doesn't mention EXT4 file systems, or what, specifically can I do to check integrity?
    – Nick
    Mar 11, 2012 at 17:16
  • Filesystem type shouldn't matter in terms of RAID integrity. If you have a maintenance period, you can unmount the affected filesystems and fsck them. If you want to check the RAID devices themselves, you can do something like echo "check" > /sys/block/md0/md/sync_action. Or echo "repair" for it to do some sort of mdadm repair.
    – cjc
    Mar 11, 2012 at 17:47
  • I'll try this tomorrow and report back.
    – Nick
    Mar 14, 2012 at 23:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .