Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have a brand new CentOS 6.5 install with two (mounted to /mnt/data) 1tb Western Digital Black drives in raid 1 with mdadm, configured via the installer. Unfortunately every now and again the entire system kernel panics with a trace similar to below:

Kernel Panic

Any tips on diagnosing or fixing this? Much appreciated!

EDIT: It appears this happened around the same time as a raid data check occured: EDIT 2: The last two crashes have happened at just past 1am Sunday, same time data check occurs.

Mar 23 01:00:02 beta kernel: md: data-check of RAID array md0
Mar 23 01:00:02 beta kernel: md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Mar 23 01:00:02 beta kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
Mar 23 01:00:02 beta kernel: md: using 128k window, over a total of 976629568k.

/proc/mdstat

    Personalities : [raid1] 
md0 : active raid1 sdc1[1] sdb1[0]
      976629568 blocks super 1.1 [2/2] [UU]
      bitmap: 0/8 pages [0KB], 65536KB chunk

unused devices: <none>

mdadm -D

/dev/md0:
        Version : 1.1
  Creation Time : Fri Mar  7 16:07:17 2014
     Raid Level : raid1
     Array Size : 976629568 (931.39 GiB 1000.07 GB)
  Used Dev Size : 976629568 (931.39 GiB 1000.07 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Sun Mar 23 03:36:59 2014
          State : active 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : beta.fmt2.spigot-servers.net:0  (local to host beta.fmt2.spigot-servers.net)
           UUID : 89a86538:f6162473:d5e0524c:b80566d6
         Events : 1728

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1

EDIT 3: Different crash, occurred during a forced resync / check, also memtest passed 4 passes just fine: http://files.md-5.net/s/X3Hi.png

EDIT 4: Even dd is causing crashes: http://files.md-5.net/s/hba2.png

EDIT 5: The SSD survives the dd torture test, guess that means I'm gonna try the drives without raid.

share|improve this question
    
I'm just trying to get my head around how the SSD relates, could you cut-and-paste the output of cat /proc/mdstat into your question? –  MadHatter Mar 23 at 9:42
    
The SSD doesn't relate. I have amended the question with the mdstat output as well as some info regarding the potential cause –  md_5 Mar 23 at 10:44
    
Yup, the last two crashes have been at just past 1am Sunday, the same time the data check is running. –  md_5 Mar 23 at 10:48
    
Try to get the first crash after a system boot. These are not the first crash and may not be indicative of the actual problem. You ought to find it in the system log /var/log/messages. Try searching the log for Not tainted. –  Michael Hampton Mar 23 at 14:17
    
The only other stuff is this: /var/log/messages-20140323:Mar 23 03:37:07 beta kernel: WARNING: at drivers/pci/dmar.c:588 warn_invalid_dmar+0x7a/0x90() (Tainted: G I--------------- ) /var/log/messages-20140323:Mar 23 03:37:07 beta kernel: Pid: 1, comm: swapper Tainted: G I--------------- 2.6.32-431.5.1.el6.x86_64 #1 Which happens right at the start of system boot indicating the bios reported DMAR at address 0. This is fine and a normal bug with Linux on all Gigabyte mobos, hasn't affected stability on any other of my systems. –  md_5 Mar 23 at 20:12

1 Answer 1

This might give an indication of the disk hardware state:

[root@ninja ~]$ /etc/rc.d/init.d/smartd start

[root@ninja ~]$ smartctl --all /dev/sdc | grep 'health'
SMART overall-health self-assessment test result: PASSED

[root@ninja ~]$ smartctl --all /dev/sdb | grep 'health'
SMART overall-health self-assessment test result: PASSED
share|improve this answer
1  
Both drives pass, they are fine. –  md_5 Mar 27 at 9:24
    
Or, indeed, smartctl -H /dev/sda. –  MadHatter Mar 27 at 9:31
1  
Have you tried other kernels, for e.g. kernel-2.6.32-358.18.1/2.6.32-431.11.2.el6? –  ALex_hha Mar 27 at 11:35

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.