Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a storage server which intermittently freezes, becomes non-responsive.

Components:

  • norco 4224
  • 14 x 2TB Western Digital Green
  • Asus p5q-e sli
  • intel core 2 duo e6700
  • 2 x Crucial Ballistix ddr2-1066 2Gb
  • OCZ Stealth-xtreme psu

Usually I do not catch it and this is a headless server sitting in the closet but when logged into an SSH session I was able to capture the following dump before it completely froze.

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.845062] Oops: 0000 [#1] SMP 

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.845065] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda/sda1/stat

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.845144] Stack:

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.845156] Call Trace:

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.845219] Code: 89 df 41 ff cf 31 c0 45 31 f6 4d 63 ff 41 bd 01 00 00 00 4d 69 ff 10 01 00 00 e8 e6 57 eb ff 49 81 c7 08 9f a6 81 45 85 e5 74 21 <49> 8b 17 48 85 d2 74 19 31 c0 48 c7 c6 41 75 77 81 48 89 df e8 

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.845253] CR2: ffffffff82371e00

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.862324] ------------[ cut here ]------------

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.862328] invalid opcode: 0000 [#2] SMP 

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.862331] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda/sda1/stat

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.862414] Stack:

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.862427] Call Trace:

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.862531] Code: be 56 eb ff eb 1e 48 8b 55 d0 48 89 df 48 c7 c6 67 7e 7a 81 31 c0 e8 a7 56 eb ff 48 8b 7d d0 e8 fd ab f2 ff 66 41 83 fc 31 76 02 <0f> 0b 45 0f b7 e4 48 c7 c6 5c a2 7a 81 48 89 df 41 ff cc 31 c0 

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.882958] ------------[ cut here ]------------

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.882962] invalid opcode: 0000 [#3] SMP 

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.882965] last sysfs file: /sys/devic es/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda/sda1/stat

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.883004] Stack:

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.883004] Call Trace:

Message from syslogd@localhost at Jun 28 16:18:39 ...
 kernel:[341209.883004] Code: be 56 eb ff eb 1e 48 8b 55 d0 48 89 df 48 c7 c6 67 7e 7a 81 31 c0 e8 a7 56 eb ff 48 8b 7d d0 e8 fd ab f2 ff 66 41 83 fc 31 76 02 <0f> 0b 45 0f b7 e4 48 c7 c6 5c a2 7a 81 48 89 df 41 ff cc 31 c0 

Message from syslogd@localhost at Jun 28 16:18:43 ...
 kernel:[341213.878511] Oops: 0011 [#4] SMP 

Message from syslogd@localhost at Jun 28 16:18:43 ...
 kernel:[341213.878513] last sysfs file: /sys/devices/virtual/block/dm-2/uevent

Message from syslogd@localhost at Jun 28 16:18:43 ...
 kernel:[341213.878588] Stack:

Message from syslogd@localhost at Jun 28 16:18:43 ...
 kernel:[341213.878600] Call Trace:

Message from syslogd@localhost at Jun 28 16:18:43 ...
 kernel:[341213.878617] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b2 5e 13 81 ff ff ff ff ba e7 3a 81 ff ff ff ff 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 

Message from syslogd@localhost at Jun 28 16:18:43 ...
 kernel:[341213.878651] CR2: ffffffff81668f70

That was the dump to my terminal before it froze. I have checked /var/log/boot.log which shows no errors on boot, /var/log/messages does not show my anything to eye catching either.

I am a developer not so much a system admin so I am not really sure how best to approach this type of intermittent failure.

If something I broken I can usually determine what and replace it to resolve hardware issues but considering this is occurring at unpredictable intervals I am having trouble assessing.

Appreciate any advice and am happy to fire the server up and post any additional information requested that may point me in the right direction!

Update:

The last time I had a session open when it froze I received a different message:

Message from syslogd@localhost at Jul 12 11:21:03 ...
kernel:[447664.831749] Bad pagetable: 001d [#2] SMP 

Message from syslogd@localhost at Jul 12 11:21:03 ...
kernel:[447664.831753] last sysfs file: /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map

Message from syslogd@localhost at Jul 12 11:21:03 ...
kernel:[447664.831749] Bad pagetable: 001d [#2] SMP 

Message fr [11:41]gd@localhost at Jul 12 11:21:03 ...
kernel:[447664.831753] last sysfs file: /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map

Update:

Abrt output for kernel oops:

[403959.325850] BUG: unable to handle kernel paging request at ffff8a0109639570
[403959.325858] IP: [<ffffffff8111846f>] atomic_long_dec_and_test+0x9/0x1a
[403959.325869] PGD 0 
[403959.325873] Oops: 0002 [#1] SMP 
[403959.325877] last sysfs file: /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map
[403959.325882] CPU 0 
[403959.325884] Modules linked in: tcp_lp fuse joydev hwmon_vid coretemp cpufreq_ondemand acpi_cpufreq freq_table mperf ipt_LOG xt_limit ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 uinput snd_hda_codec_via snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer matroxfb_base matroxfb_DAC1064 matroxfb_accel matroxfb_Ti3026 matroxfb_g450 ppdev parport_pc r8169 snd soundcore snd_page_alloc iTCO_wdt g450_pll parport mii serio_raw asus_atk0110 microcode i2c_i801 i2c_core matroxfb_misc iTCO_vendor_support pata_acpi ata_generic pata_jmicron mvsas libsas sata_sil24 scsi_transport_sas [last unloaded: scsi_wait_scan]
[403959.325944] 
[403959.325949] Pid: 1937, comm: sendmail Not tainted 2.6.35.13-92.fc14.x86_64 #1 P5QL/EPU/System Product Name
[403959.325953] RIP: 0010:[<ffffffff8111846f>]  [<ffffffff8111846f>] atomic_long_dec_and_test+0x9/0x1a
[403959.325960] RSP: 0018:ffff880109637988  EFLAGS: 00010292
[403959.325964] RAX: ffff88010be60880 RBX: ffff8a0109639540 RCX: dead000000100100
[403959.325968] RDX: dead000000200200 RSI: 0000000000000296 RDI: ffff8a0109639570
[403959.325971] RBP: ffff880109637988 R08: ffff880109637b20 R09: 00000000004c4b3f
[403959.325975] R10: 00016f661b5b44f1 R11: ffffffff81b81ee0 R12: 0000000000000001
[403959.325979] R13: ffff880109637ac8 R14: ffff880109637ac8 R15: 0000000000000000
[403959.325983] FS:  00007fbf2b91d7e0(0000) GS:ffff880002000000(0000) knlGS:0000000000000000
[403959.325987] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[403959.325991] CR2: ffff8a0109639570 CR3: 0000000109735000 CR4: 00000000000006f0
[403959.325995] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[403959.325998] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[403959.326003] Process sendmail (pid: 1937, threadinfo ffff880109636000, task ffff88011a0f9740)
[403959.326006] Stack:
[403959.326008]  ffff8801096379d8 ffffffff81118985 ffff8801096379a8 ffffffff8146ac67
[403959.326014] <0> ffff8801096379d8 0000000000000000 0000000000000001 ffff880109637ac8
[403959.326020] <0> ffff880109637ac8 0000000000000000 ffff880109637a08 ffffffff81125bb7
[403959.326027] Call Trace:
[403959.326033]  [<ffffffff81118985>] fput+0x22/0x1ed
[403959.326040]  [<ffffffff8146ac67>] ? _raw_spin_unlock_irqrestore+0x17/0x19
[403959.326046]  [<ffffffff81125bb7>] poll_freewait+0x45/0x94
[403959.326051]  [<ffffffff81126314>] do_select+0x50f/0x52c
[403959.326056]  [<ffffffff81125c06>] ? __pollwait+0x0/0xcc
[403959.326061]  [<ffffffff81125cd2>] ? pollwake+0x0/0x54
[403959.326070]  [<ffffffff811ded91>] ? avc_has_perm+0x5c/0x6e
[403959.326079]  [<ffffffff8103c1a5>] ? need_resched+0x23/0x2d
[403959.326084]  [<ffffffff81106605>] ? __raw_local_irq_save+0x1d/0x23
[403959.326091]  [<ffffffff81130238>] ? seq_open+0x24/0xab
[403959.326098]  [<ffffffff8121eb60>] ? put_dec+0x20/0x3c
[403959.326103]  [<ffffffff8121fcbe>] ? number.clone.1+0x138/0x24c
[403959.326109]  [<ffffffff81109df0>] ? kmem_cache_alloc_notrace+0x8b/0xb1
[403959.326114]  [<ffffffff81220920>] ? vsnprintf+0x83/0x44c
[403959.326119]  [<ffffffff8103c1a5>] ? need_resched+0x23/0x2d
[403959.326124]  [<ffffffff8103c1bd>] ? should_resched+0xe/0x2e
[403959.326129]  [<ffffffff81469699>] ? _cond_resched+0xe/0x22
[403959.326134]  [<ffffffff81125844>] ? might_fault+0x21/0x23
[403959.326138]  [<ffffffff81106605>] ? __raw_local_irq_save+0x1d/0x23
[403959.326143]  [<ffffffff811264a5>] core_sys_select+0x174/0x213
[403959.326151]  [<ffffffff810a9da9>] ? check_for_new_grace_period.clone.19+0x8b/0x97
[403959.326157]  [<ffffffff810a9ecc>] ? __call_rcu+0x117/0x120
[403959.326164]  [<ffffffff81052dd2>] ? timespec_add_safe+0x37/0x65
[403959.326172]  [<ffffffff810102e4>] ? read_tsc+0x9/0x19
[403959.326178]  [<ffffffff8106e4ef>] ? ktime_get_ts+0x85/0x8e
[403959.326183]  [<ffffffff811265da>] sys_select+0x96/0xbe
[403959.326189]  [<ffffffff81009cf2>] system_call_fastpath+0x16/0x1b
[403959.326192] Code: c7 48 83 c4 28 4c 89 f8 5b 41 5c 41 5d 41 5e 41 5f c9 c3 90 55 48 89 e5 0f 1f 44 00 00 f0 ff 07 c9 c3 55 48 89 e5 0f 1f 44 00 00 <f0> 48 ff 0f 0f 94 c0 84 c0 c9 0f 95 c0 0f b6 c0 c3 55 48 89 e5 
[403959.326241] RIP  [<ffffffff8111846f>] atomic_long_dec_and_test+0x9/0x1a
[403959.326247]  RSP <ffff880109637988>
[403959.326249] CR2: ffff8a0109639570
share|improve this question
    
What does abrt say? Checked the memory? –  phresus Jul 12 '11 at 16:50
    
Have you applied all the latest kernel/software patches to your system? (Sorry, I hate when vendors do that to me, but it's a good starting point...perhaps your problem is resolved with more recent kernel patches, device drivers, etc...). –  loopforever Jul 12 '11 at 22:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.