2

I have a new Ubuntu Server (11.04) that keeps crashing, especially during heavy disk I/O (like making a backup). It's drives are configures as a RAID 10 with 4 1TB Western Digital Caviar Black Hard Drives.

The message I'm seeing via /proc/kmsg when it crashes is, "failed command: READ FPDMA QUEUED".

This seems like something is messed up with either the drives or the software raid is broken?

The machine was doing fine until this afternoon when it crashed during a file transfer Ever since then it's been crashing when I try to run a backup, but it's not always the same file or place each time.

How do I know if it's a software or hardware failure? How do I know if it's the SATA controller or one of the disks?

Also, all 4 drives in the array "completed without error" when I ran an extended off-line test on them.

This is the full output of /proc/kmsg from the time I rebooted until it crashed again:

[  356.076292] type=1400 audit(1311983491.536:14): apparmor="DENIED" operation="open" parent=1397 profile="/usr/lib/libvirt/virt-aa-helper" name="/dev/dm-9" pid=2222 comm="virt-aa-helper" requested_mask="r" denied_mask="r" fsuid=0 ouid=105
[  356.304840] type=1400 audit(1311983491.766:15): apparmor="STATUS" operation="profile_load" name="libvirt-c67f4a48-2cad-6deb-d7e7-13f9c7620ad9" pid=2223 comm="apparmor_parser"
[  357.002246] device vnet0 entered promiscuous mode
[  357.003702] br0: port 2(vnet0) entering learning state
[  357.003704] br0: port 2(vnet0) entering learning state
[  366.020017] br0: port 2(vnet0) entering forwarding state
[  367.050024] vnet0: no IPv6 routers present
 l)idx08f5
1[20.298 aefas x0000000(
4[20.294 i:38 om d_ad0Nttitd263-0sre 4-bnu 15054]Cl rc: 15055]  15055]  15055]  15055]  15055]  15056]   1bfpo_re05/x0[ad0
4[20.292 [ffff81de>  epo_re05/x0 15057]   ycrqetwie03b040[ad0
4[20.290 [ffffa017>  ad0+xe/x5 ri1] 15058]   eal_pnlc_lg+x/x0 15058]  15059]  15059]  15059]  15059]  15059]   enltra_epr0001
4[20.200 ialn okdbgigdet enltit_ x0000 H D hne
3[55.640 t4 Err  eoCm ess HRyh 08 
3[55.646 t40:fie omn:RA PM UUD 84088]aa.0 m 00:00:1d/40:90:04 a  c 228i
3[55.644     rs4/05:01:a0:03:00/0Eak01 AAbserr
3[55.643 t40:sau:{DD 
3[55.647 t40:fie omn:RA PM UUD 84080]aa.0 m 00:80:dd/40:90:04 a  c 228i
3[55.654     rs4/05:01:a0:03:00/0Eak01 AAbserr
3[55.653 t40:sau:{DD 
3[55.656 t40:fie omn:RA PM UUD 84082]aa.0 m 00:00:5d/40:90:04 a  c 228i
3[55.653     rs4/05:01:a0:03:00/0Eak01 AAbserr
3[55.652 t40:sau:{DD 
3[55.656 t40:fie omn:RA PM UUD 84084]aa.0 m 00:80:9d/40:90:04 a  c 228i
3[55.652     rs4/05:01:a0:03:00/0Eak01 AAbserr
3[55.651 t40:sau:{DD 
3[55.655 t40:fie omn:RA PM UUD 84086]aa.0 m 00:00:dd/40:90:04 a  c 228i
3[55.651     rs4/05:01:a0:03:00/0Eak01 AAbserr
3[55.650 t40:sau:{DD 
3[55.654 t40:fie omn:RA PM UUD 84088]aa.0 m 00:80:1d/40:90:04 a  c 228i
3[55.650     rs4/05:01:a0:03:00/0Eak01 AAbserr
3[55.650 t40:sau:{DD 
3[55.654 t40:fie omn:RA PM UUD 84089]aa.0 m 00:00:5d/40:90:04 a  c 228i
3[55.660     rs4/05:01:a0:03:00/0Eak01 AAbserr
3[55.668 t40:sau:{DD 
3[55.662 t40:fie omn:RA PM UUD 84081]aa.0 m 00:80:9d/40:90:04 a  c 228i
3[55.668     rs4/05:01:a0:03:00/0Eak01 AAbserr
3[55.668 t40:sau:{DD 
3[55.724 t40:fie omn:RA PM UUD:03:00/0tg8nq548 n 84008]     e 00:c0:9d/00:90:04 ms x0(T u ro) 84010]aa.0 tts  RY} 84029]aa.0 aldcmad EDFDAQEE
3[55.706 t40:cd6/04:01:a0:03:00/0tg9nq548 n 84034]     e 00:c0:9d/00:90:04 ms x0(T u ro) 84049]aa.0 tts  RY} 84048]aa.0 aldcmad EDFDAQEE
3[55.739 t40:cd6/05:01:a0:03:00/0tg1 c 228i
3[55.739     rs4/05:01:a0:03:00/0Eak01 AAbserr
3[55.742 t40:sau:{DD 
3[55.703 t40:fie omn:RA PM UUD 84074]aa.0 m 00:80:9d/40:90:04 a 1nq548 n 84074]     e 00:c0:9d/00:90:04 ms x0(T u ro) 84083]aa.0 tts  RY} 84098]aa.0 aldcmad EDFDAQEE
3[55.797 t40:cd6/06:01:a0:03:00/0tg1 c 228i
3[55.798     rs4/05:01:a0:03:00/0Eak01 AAbserr
3[55.811 t40:sau:{DD 
0 aldcmad EDFDAQEE
3[55.839 t40:cd6/06:0a:90:03:00/0tg1 c 228i
3[55.830     rs4/05:01:a0:03:00/0Eak01 AAbserr
3[55.851 t40:sau:{DD 
3[55.818 t40:fie omn:RA PM UUD 84048]aa.0 m 00:00:9d/40:90:04 a 4nq548 n 84048]     e 00:c0:9d/00:90:04 ms x0(T u ro) 84058]aa.0 tts  RY} 84064]aa.0 aldcmad EDFDAQEE
3[55.809 t40:cd6/07:0a:90:03:00/0tg1 c 228i
3[55.800     rs4/05:01:a0:03:00/0Eak01 AAbserr
3[55.826 t40:sau:{DD 
3[55.871 t40:fie omn:RA PM UUD 84098]aa.0 m 00:00:1d/40:90:04 a 6nq548 n 84098]     e 00:c0:9d/00:90:04 ms x0(T u ro) 84007]aa.0 tts  RY}[ 5854.091012] ata4.00: failed command: READ FPDMA QUEUED
[ 5854.091573] ata4.00: cmd 60/00:88:00:b5:d9/04:00:39:00:00/40 tag 17 ncq 524288 in
[ 5854.091574]          res 40/00:5c:00:19:da/00:00:39:00:00/40 Emask 0x10 (ATA bus error)
[ 5854.092690] ata4.00: status: { DRDY }
[ 5854.093287] ata4.00: failed command: READ FPDMA QUEUED
[ 5854.093817] ata4.00: cmd 60/00:90:00:b9:d9/04:00:39:00:00/40 tag 18 ncq 524288 in
[ 5854.093817]          res 40/00:5c:00:19:da/00:00:39:00:00/40 Emask 0x10 (ATA bus error)
[ 5854.094964] ata4.00: status: { DRDY }
[ 5854.095510] ata4.00: failed command: READ FPDMA QUEUED
[ 5854.096105] ata4.00: cmd 60/00:98:00:bd:d9/04:00:39:00:00/40 tag 19 ncq 524288 in
[ 5854.096105]          res 40/00:5c:00:19:da/00:00:39:00:00/40 Emask 0x10 (ATA bus error)
[ 5854.097246] ata4.00: status: { DRDY }
[ 5854.097773] ata4.00: failed command: READ FPDMA QUEUED
[ 5854.098338] ata4.00: cmd 60/00:a0:00:c1:d9/04:00:39:00:00/40 tag 20 ncq 524288 in
[ 5854.098339]          res 40/00:5c:00:19:da/00:00:39:00:00/40 Emask 0x10 (ATA bus error)
[ 5854.099491] ata4.00: status: { DRDY }
[ 5854.100084] ata4.00: failed command: READ FPDMA QUEUED
[ 5854.100658] ata4.00: cmd 60/00:a8:00:c5:d9/04:00:39:00:00/40 tag 21 ncq 524288 in
[ 5854.100659]          res 40/00:5c:00:19:da/00:00:39:00:00/40 Emask 0x10 (ATA bus error)
[ 5854.101775] ata4.00: status: { DRDY }
[ 5854.102392] ata4.00: failed command: READ FPDMA QUEUED
[ 5854.102924] ata4.00: cmd 60/00:b0:00:c9:d9/04:00:39:00:00/40 tag 22 ncq 524288 in
[ 5854.102925]          res 40/00:5c:00:19:da/00:00:39:00:00/40 Emask 0x10 (ATA bus error)
[ 5854.104080] ata4.00: status: { DRDY }
[ 5854.104639] ata4.00: failed command: READ FPDMA QUEUED
[ 5854.105206] ata4.00: cmd 60/00:b8:00:cd:d9/04:00:39:00:00/40 tag 23 ncq 524288 in
[ 5854.105207]          res 40/00:5c:00:19:da/00:00:39:00:00/40 Emask 0x10 (ATA bus error)
[ 5854.106377] ata4.00: status: { DRDY }
[ 5854.106921] ata4.00: failed command: READ FPDMA QUEUED
[ 5854.107493] ata4.00: cmd 60/00:c0:00:d1:d9/04:00:39:00:00/40 tag 24 ncq 524288 in
[ 5854.107493]          res 40/00:5c:00:19:da/00:00:39:00:00/40 Emask 0x10 (ATA bus error)
[ 5854.108651] ata4.00: status: { DRDY }
[ 5854.109201] ata4.00: failed command: READ FPDMA QUEUED
[ 5854.109784] ata4.00: cmd 60/00:c8:00:d5:d9/04:00:39:00:00/40 tag 25 ncq 524288 in
[ 5854.109785]          res 40/00:5c:00:19:da/00:00:39:00:00/40 Emask 0x10 (ATA bus error)
[ 5854.110981] ata4.00: status: { DRDY }
[ 5854.111555] ata4.00: failed command: READ FPDMA QUEUED
[ 5854.112111] ata4.00: cmd 60/00:d0:00:d9:d9/04:00:39:00:00/40 tag 26 ncq 524288 in
[ 5854.112111]          res 40/00:5c:00:19:da/00:00:39:00:00/40 Emask 0x10 (ATA bus error)
[ 5854.113248] ata4.00: status: { DRDY }
[ 5854.113839] ata4.00: failed command: READ FPDMA QUEUED
[ 5854.114408] ata4.00: cmd 60/00:d8:00:dd:d9/04:00:39:00:00/40 tag 27 ncq 524288 in
[ 5854.114409]          res 40/00:5c:00:19:da/00:00:39:00:00/40 Emask 0x10 (ATA bus error)
[ 5854.115586] ata4.00: status: { DRDY }
[ 5854.116127] ata4.00: failed command: READ FPDMA QUEUED
[ 5854.116691] ata4.00: cmd 60/00:e0:00:e1:d9/04:00:39:00:00/40 tag 28 ncq 524288 in
[ 5854.116691]          res 40/00:5c:00:19:da/00:00:39:00:00/40 Emask 0x10 (ATA bus error)
[ 5854.117856] ata4.00: status: { DRDY }
[ 5854.118430] ata4.00: failed command: READ FPDMA QUEUED
[ 5854.119004] ata4.00: cmd 60/00:e8:00:e5:d9/04:00:39:00:00/40 tag 29 ncq 524288 in
[ 5854.119005]          res 40/00:5c:00:19:da/00:00:39:00:00/40 Emask 0x10 (ATA bus error)
[ 5854.120213] ata4.00: status: { DRDY }
[ 5855.100038] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 5855.104300] ata4.00: configured for UDMA/133
[ 5855.104351] ata4: EH complete
[10013.907683] general protection fault: 0000 [#1] SMP 
[10013.907997] last sysfs file: /sys/devices/system/cpu/cpu5/cache/index2/shared_cpu_map
[10013.908574] CPU 0 
[10013.908577] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables kvm_amd kvm eeepc_wmi sparse_keymap bridge stp nouveau sp5100_tco ttm i2c_piix4 edac_core edac_mce_amd drm_kms_helper k10temp drm i2c_algo_bit video lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov usb_storage uas r8169 xhci_hcd ahci libahci raid6_pq async_tx raid1 raid0 multipath linear
[10013.911080] 
[10013.911734] Pid: 349, comm: md0_resync Tainted: G    B       2.6.38-10-server #46-Ubuntu To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX
[10013.912418] RIP: 0010:[]  [] kmem_cache_alloc+0x58/0x110
[10013.913102] RSP: 0018:ffff88041b6079c0  EFLAGS: 00010006
[10013.913783] RAX: 0000000000000000 RBX: ffff88043f802600 RCX: ffffffff813df60a
[10013.914472] RDX: 0000000000000000 RSI: 0000000000000020 RDI: ffff88043f802600
[10013.915159] RBP: ffff88041b607a00 R08: ffff8800bd416a80 R09: ffff880414db0500
[10013.915839] R10: 00000000684eb800 R11: 0000000000000001 R12: 0200000000000000
[10013.916523] R13: 0000000000000086 R14: 0000000000000020 R15: ffff88041b42b400
[10013.917204] FS:  00007f741ce31700(0000) GS:ffff8800bd400000(0000) knlGS:0000000000000000
[10013.917529] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[10013.917529] CR2: 00007f4402d5dd00 CR3: 00000004108a5000 CR4: 00000000000006f0
[10013.917529] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[10013.917529] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[10013.917529] Process md0_resync (pid: 349, threadinfo ffff88041b606000, task ffff88041b945b80)
[10013.917529] Stack:
[10013.917529]  ffff88041b6079e0 000000201b6079e0 0000000000000082 ffffffff81a77fa0
[10013.917529]  ffff880414db0500 0000000000000020 0000000000000020 ffff88041b42b400
[10013.917529]  ffff88041b607a30 ffffffff813df60a ffff880400000000 ffff88041be74020
[10013.917529] Call Trace:
[10013.917529]  [] scsi_pool_alloc_command+0x4a/0x80
[10013.917529]  [] scsi_host_alloc_command.clone.7+0x33/0xa0
[10013.159 [ffff83ff> _cigtcmad02/x0103972]  cistpf_md08/x0103972] 103972] 103972] 103972]  _eei_nlgdvc+x704
4[01.159 [ffff822c> eei_nlgdvc+x005
4[01.159 [ffff82c9> l_nlg02/x0103972]  ad0upu+xd03 ri1]103972]  dupu+x204
4[01.159 [ffff8421> dd_yc08d0c0103972] [10013.917529]  [] ? _raw_spin_lock+0xe/0x20
[10013.917529]  [] ? recalc_sigpending+0x1/x0103972] 103972] 103972] 103972] 103972]   enltra_epr0001
0[01.159 oe 66 06 04 9c a6 69 66 04 b0 54 30 5a c0 04 b2 d8 40 4a 00 04 34 8103972]RP 103972] S ff80167c>103972]-- n rc 5355716c]-

This is the output from syslog:

Jul 29 21:23:09 Othelo kernel: [ 5854.068449] ata4.00: exception Emask 0x10 SAct 0x7fffffff SErr 0x90202 action 0xe frozen
Jul 29 21:23:09 Othelo kernel: [ 5854.068464] ata4.00: irqstat00400,PYRYcagd<> 84087]aa:Sro:{RcvomPritPYdCg1BB}<> 84087]aa.0 aldcmad EDFDAQEE
Jul 29 21:23:09 Othelo kernel: 3[55.643 t40:cd6/00:0f:90:03:00/0tg0nq548 n<> 84088]     e 00:c0:9d/00:90:04 ms x0(T u ro)<> 84089]aa.0 tts  RY}<> 84089]aa.0 aldcmad EDFDAQEE
Jul 29 21:23:09 Othelo kernel: 3[55.653 t40:cd6/00:0e:90:03:00/0tg1nq548 n<> 84080]     e 00:c0:9d/00:90:04 ms x0(T u ro)<> 84081]aa.0 tts  RY}<> 84081]aa.0 aldcmad EDFDAQEE
Jul 29 21:23:09 Othelo kernel: 3[55.652 t40:cd6/01:0f:90:03:00/0tg2nq548 n<> 84082]     e 00:c0:9d/00:90:04 ms x0(T u ro)<> 84083]aa.0 tts  RY}<> 84083]aa.0 aldcmad EDFDAQEE
Jul 29 21:23:09 Othelo kernel: 3[55.652 t40:cd6/01:0f:90:03:00/0tg3nq548 n<> 84084]     e 00:c0:9d/00:90:04 ms x0(T u ro)<> 84085]aa.0 tts  RY}<> 84085]aa.0 aldcmad EDFDAQEE
Jul 29 21:23:09 Othelo kernel: 3[55.651 t40:cd6/02:0f:90:03:00/0tg4nq548 n<> 84086]     e 00:c0:9d/00:90:04 ms x0(T u ro)<> 84087]aa.0 tts  RY}<> 84087]aa.0 aldcmad EDFDAQEE
Jul 29 21:23:09 Othelo kernel: 3[55.650 t40:cd6/02:00:a0:03:00/0tg5nq548 n<> 84088]     e 00:c0:9d/00:90:04 ms x0(T u ro)<> 84089]aa.0 tts  RY}<> 84089]aa.0 aldcmad EDFDAQEE
Jul 29 21:23:09 Othelo kernel: 3[55.659 t40:cd6/03:00:a0:03:00/0tg6nq548 n<> 84080]     e 00:c0:9d/00:90:04 ms x0(T u ro)<> 84080]aa.0 tts  RY}<> 84081]aa.0 aldcmad EDFDAQEE
Jul 29 21:23:09 Othelo kernel: 3[55.667 t40:cd6/03:00:a0:03:00/0tg7nq548 n<> 84081]     e 00:c0:9d/00:90:04 ms x0(T u ro)<> 84099]aa.0 tts  RY}<> 84007]aa.0 aldcmad EDFDAQEE
Jul 29 21:23:09 Othelo kernel: [ 5854.070785] ata4.00: cmd 60/00:40:00:0d:da/040:90:04 a  c 228i
Jul 29 21:23:09 Othelo kernel: 3[55.776     rs4/05:01:a0:03:00/0Eak01 AAbserr
Jul 29 21:23:09 Othelo kernel: 3[55.797 t40:sau:{DD 
Jul 29 21:23:09 Othelo kernel: 3[55.743 t40:fie omn:RA PM UUD<> 84034]aa.0 m 00:80:1d/40:90:04 a  c 228i
Jul 29 21:23:09 Othelo kernel: 3[55.707     rs4/05:01:a0:03:00/0Eak01 AAbserr
Jul 29 21:23:09 Othelo kernel: 3[55.710 t40:sau:{DD 
Jul 29 21:23:09 Othelo kernel: 3[55.777 t40:fie omn:RA PM UUD<> 84052]aa.0 m 00:00:5d/40:90:04 a 0nq548 n<> 84052]     e 00:c0:9d/00:90:04 ms x0(T u ro)<> 84069]aa.0 tts  RY}<> 84075]aa.0 aldcmad EDFDAQEE
Jul 29 21:23:09 Othelo kernel: 3[55.763 t40:cd6/05:01:a0:03:00/0tg1 c 228i
Jul 29 21:23:09 Othelo kernel: 3[55.763     rs4/05:01:a0:03:00/0Eak01 AAbserr
Jul 29 21:23:09 Othelo kernel: 3[55.784 t40:sau:{DD 
Jul 29 21:23:09 Othelo kernel: 3[55.737 t40:fie omn:RA PM UUD<> 84096]aa.0 m 00:00:dd/40:90:04 a 2nq548 n<> 84096]     e 00:c0:9d/00:90:04 ms x0(T u ro)<> 84018]aa.0 tts  RY}<3>[ 5854.081762] ata4.0:fie omn:RA PM UUD<> 84021]aa.0 m 00:80:5d/40:90:04 a 3nq548 n<> 84022]     e 00:c0:9d/00:90:04 ms x0(T u ro)<> 84032]aa.0 tts  RY}<> 84045]aa.0 aldcmad EDFDAQEE
Jul 29 21:23:09 Othelo kernel: 3[55.867 t40:cd6/07:0a:90:03:00/0tg1 c 228i
Jul 29 21:23:09 Othelo kernel: 3[55.868     rs4/05:01:a0:03:00/0Eak01 AAbserr
Jul 29 21:23:09 Othelo kernel: 3[55.881 t40:sau:{DD 
Jul 29 21:23:09 Othelo kernel: 3[55.849 t40:fie omn:RA PM UUD<> 84073]aa.0 m 00:80:dd/40:90:04 a 5nq548 n<> 84074]     e 00:c0:9d/00:90:04 ms x0(T u ro)<> 84082]aa.0 tts  RY}<> 84083]aa.0 aldcmad EDFDAQEE
Jul 29 21:23:09 Othelo kernel: 3[55.822 t40:cd6/08:0b:90:03:00/0tg1 c 228i
Jul 29 21:23:09 Othelo kernel: 3[55.823     rs4/05:01:a0:03:00/0Eak01 AAbserr
Jul 29 21:23:09 Othelo kernel: 3[55.943 t40:sau:{DD 
Jul 29 21:23:09 Othelo kernel: [ 5854.120747] ata4.00: failed command: READ FPDMA QUEUED
Jul 29 21:23:09 Othelo kernel: [ 5854.121321] ata4.00: cmd 60/00:f0:00:e9:d9/04:00:39:00:00/40 tag 30 ncq 524288 in
Jul 29 21:23:09 Othelo kernel: [ 5854.121322]          res 40/00:5c:00:19:da/00:00:39:00:00/40 Emask 0x10 (ATA bus error)
Jul 29 21:23:09 Othelo kernel: [ 5854.122483] ata4.00: status: { DRDY }
Jul 29 21:23:09 Othelo kernel: [ 5854.123050] ata4: hard resetting link
Jul 29 21:42:45 Othelo mdadm[1563]: Rebuild62 event detected on md device /dev/md/0
Jul 29 22:16:06 Othelo mdadm[1563]: Rebuild81 event detected on md device /dev/md/0
Jul 29 22:17:01 Othelo CRON[13235]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 29 22:32:29 Othelo kernel: 4972] <ffff1d8e]_ss_e_omn+xe0c
Jul 29 22:32:29 Othelo kernel: 4[01.159 [ffff83fd> cigtcmad04/x0<>103972] <ffff1efd]ss_eu_scn+xd0e
Jul 29 22:32:29 Othelo kernel: 4[01.159 [ffff843a> dpe_n0a/xa
Jul 29 22:32:29 Othelo kernel: 4[01.159 [ffff821b>  ediemv_eus+x80a
Jul 29 22:32:29 Othelo kernel: 4[01.159 [ffff8222> l_ekrqet0c/x1
Jul 29 22:32:29 Othelo kernel: 4[01.159 [ffff837c> cirqetf+x6040<>103972] <ffff1c77]_gnrcupu_eie03/x0<>103972] <ffff1ca0]gnrcupu_eie03/x0<>103972] <ffff1b75]bkupu+x506
Jul 29 22:32:29 Othelo kernel: 4[01.159 [ffffa0f8> nlgsae+x00c ri1]<>103972] <ffff0ead]ri1_nlg01/x0[ad0
Jul 29 22:32:29 Othelo kernel: 4[01.159 [ffff82c9> l_nlg02/x0<>103972] <ffff19f2]m_nlg02/x0<>103972] <ffff198d]m_osn+x4/x9
Jul 29 22:32:29 Othelo kernel: 4[01.159 [ffff807c>  uoeb05
Jul 29 22:32:29 Othelo kernel: 4[01.159 [ffff84ca> dtra+x1/x5
Jul 29 22:32:29 Othelo kernel: 4[01.159 [ffff84c9>  dtra+x/x5
Jul 29 22:32:29 Othelo kernel: 4[01.159 [ffff8077> tra+x60a
Jul 29 22:32:29 Othelo kernel: 4[01.159 [ffff80ce> enltra_epr0401
Jul 29 22:32:29 Othelo kernel: 4[01.159 [ffff807e>  tra+x/x0<>103972] <ffff10d0]?kre_hedhle+x/x0<>103972]Cd:6 69 69 98 5f 66 06 69 c8 76 c0 42 8d 00 d8 04 5e f8 00 00 86 71 4>8 40 98 04 9e 79 66 06 04 5e 5
Jul 29 22:32:29 Othelo kernel: 1[01.159 I [ffff815e> mmccealc05/x1
Jul 29 22:32:29 Othelo kernel: 4[01.159 RP<ff84b090


UPDATE: Checked all Hard drives by running full scan on each and writing zeros to all drives. No problems found. I'm going to reinstall everything and try again. :/

4 Answers 4

4

You answer your own question by mentioning changing a cable, zeroing the hard drives and starting from scratch seem to have fixed the problem, but to me it sounds like you could be having a case of the 550559 curse.

Therefore, I am wondering if you could comment the issue, whether it has returned or not, and if so how you fixed it the second time.

I don't have enough rep to comment on your answer, so that's why I'll provide this feedback in the form of an answer: Could be Bug #550559. 5 releases of Ubuntu later, still unassigned.

If so, tagging the bug with 'This bug affects you' would help make the bug more prone to be fixed.

7
  • It crashed again last night so I'm un-checking my previous answer for now. 4 months of uptime isn't good enough. I couldn't tell what exactly that bug was about since it seemed rather vague to me. I'm wondering if its time to try something else like Cent OS or Debian Stable, if maybe those would be more reliable?
    – Nick
    Feb 5, 2012 at 22:37
  • 1
    Do you know if bug 550559 affects Debian Stable, ie. 6.04?
    – Nick
    Feb 6, 2012 at 3:47
  • As follow-up, I installed CentOS 6.2 Minimal on the same hardware and has been running for one month. I guess in a year or so we'll know if it was OS related!
    – Nick
    Apr 20, 2012 at 15:28
  • Still having this issue. New motherboard, new hard drives, new SATA cables, new OS... But on CentOS 6.2, when this error occurs it doesn't Kernel Panic the system, it just seems to reset the SATA link and try again. I'm wondering if this issue might be related to RAID 10 on a bus that can handle 6Gbs on a single drive but not 6Gbs on 4 drives at the same time. Looking at the CentOS logs, eventually it drops the link speed to 3Gbs and resets and that drive doesn't have any more errors for the session.
    – Nick
    Jun 19, 2012 at 22:04
  • Hi Nick and sorry for not getting back to you, my spamfilter ate my stack notifications. I've seen this trouble, whether it's the bug or not, being very much alive here and there. It's not even necessarily linked to complex RAID setups. It's still seems to happen at multiple random combinations of factors, and no educated guesses yet to what is the common denominator in these reports. Unfortunately, I cannot be of any more help. In my case, 550559 didn't return since I replaced sata-cables. But looking at people who replaced all hardware, I guess this just proves I am one of the lucky ones.
    – Redsandro
    Jun 20, 2012 at 11:50
3

You may want to try updating your BIOS firmware. I had the exact same error, updated to latest BIOS, and the problem disappeared.

1

Whichever drive is connected to SATA port 4 is having problems. It's not necessarily that the drive itself is cactus, it could be a cable or even the SATA port. Given that the server is new, I'd be drawing on the system warranty to replace the whole thing; it's not worth your time to diagnose piddling problems like this -- that's why you always buy decent systems with a good warranty.

4
  • Does port 4 always map to /dev/sdd? Or is there a command to see which /dev device is which SATA port? I have warranties on parts but not the whole system since we build them in house. I think I'll try swapping the SATA cables to see if that's the problem.
    – Nick
    Jul 30, 2011 at 4:27
  • No, the mapping between drives and SATA ports is irritatingly inconsistent. You should be able to get the mapping out of your boot messages, though.
    – womble
    Jul 30, 2011 at 4:47
  • After swapping the cables I got it to crash again, but this time the messages didn't seem hard drive related. I'm wondering if maybe it was a loose cable initially, but having to hard reset caused other corruption? I'd like to run some kind of complete and thorough read-write test on each drive individually to rule out any problems. Would using a utility to write zeros to the whole drive report errors if there were any? Would it have an error if the cable had a problem? Is there another (better) way to thoroughly test a SATA drive (even if the test is destructive to data)?
    – Nick
    Jul 31, 2011 at 22:27
  • 1
    Questions go in questions, not comments, because hardly anybody reads comments.
    – womble
    Jul 31, 2011 at 22:57
1

I think this issue was due to software corruption. Zeroing the hard drives and starting from scratch seems to have fixed the problem, and the system now appears to be stable.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .