Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm running Ubuntu 10.10 on a webserver, and found that apache wasn't responding anymore. I checked, and got this from /var/log/messages:

Jul 31 07:42:11 web1 kernel: [1673445.610690] CPU 2
Jul 31 07:42:11 web1 kernel: [1673445.610696] Modules linked in: parport_pc ppdev snd_hda_codec_atihdmi snd_hda_intel radeon snd_hda_codec snd_hwdep ttm drm_kms_helper drm i7core_edac snd_pcm snd_timer i2c_algo_bit snd soundcore edac_core snd_page_alloc lp parport xfs exportfs multipath linear aacraid 3w_9xxx 3w_xxxx raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx raid0 raid1 ahci r8169 libahci mii sata_nv sata_sil sata_via
Jul 31 07:42:11 web1 kernel: [1673445.610931]
Jul 31 07:42:11 web1 kernel: [1673445.610950] Pid: 27609, comm: apache2 Not tainted 2.6.35-30-server #54-Ubuntu MSI X58 Pro-E (MS-7522)/MS-7522
Jul 31 07:42:11 web1 kernel: [1673445.610998] RIP: 0010:[<ffffffff8104dd2a>]  [<ffffffff8104dd2a>] task_rq_lock+0x4a/0xa0
Jul 31 07:42:11 web1 kernel: [1673445.611047] RSP: 0018:ffff8805a43f3dc8  EFLAGS: 00010082
Jul 31 07:42:11 web1 kernel: [1673445.611073] RAX: 909090909005eb05 RBX: 0000000000015b80 RCX: 0000000000000000
Jul 31 07:42:11 web1 kernel: [1673445.611115] RDX: 0000000000000282 RSI: ffff8805a43f3e20 RDI: 00007fcd07e04b40
Jul 31 07:42:11 web1 kernel: [1673445.611157] RBP: ffff8805a43f3de8 R08: 0000000000000000 R09: dead000000200200
Jul 31 07:42:11 web1 kernel: [1673445.611199] R10: dead000000100100 R11: dead000000200200 R12: 00007fcd07e04b40
Jul 31 07:42:11 web1 kernel: [1673445.611241] R13: ffff8805a43f3e20 R14: 0000000000015b80 R15: 0000000000000002
Jul 31 07:42:11 web1 kernel: [1673445.611283] FS:  00007fcd0889d740(0000) GS:ffff880001e40000(0000) knlGS:0000000000000000
Jul 31 07:42:11 web1 kernel: [1673445.611327] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 31 07:42:11 web1 kernel: [1673445.611353] CR2: 00007fcd0a936bb0 CR3: 00000005a4055000 CR4: 00000000000006e0
Jul 31 07:42:11 web1 kernel: [1673445.611395] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jul 31 07:42:11 web1 kernel: [1673445.611437] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jul 31 07:42:11 web1 kernel: [1673445.611480] Process apache2 (pid: 27609, threadinfo ffff8805a43f2000, task ffff8803e2308000)
Jul 31 07:42:11 web1 kernel: [1673445.611543]  00007fcd07e04b40 ffff8802fa6ffec8 000000000000000f 0000000000000000
Jul 31 07:42:11 web1 kernel: [1673445.611576] <0> ffff8805a43f3e58 ffffffff8105895c ffff8803e40ff550 ffff880001e50ea0
Jul 31 07:42:11 web1 kernel: [1673445.611626] <0> ffffffff81a44700 0000000000000286 ffffffff81a52850 0000000000000282
Jul 31 07:42:11 web1 kernel: [1673445.611713]  [<ffffffff8105895c>] try_to_wake_up+0x3c/0x400
Jul 31 07:42:11 web1 kernel: [1673445.611739]  [<ffffffff81058d75>] wake_up_process+0x15/0x20
Jul 31 07:42:11 web1 kernel: [1673445.611767]  [<ffffffff812531b0>] freeary+0x1e0/0x220
Jul 31 07:42:11 web1 kernel: [1673445.611793]  [<ffffffff81254220>] T.579+0xb0/0x100
Jul 31 07:42:11 web1 kernel: [1673445.611819]  [<ffffffff815a6a69>] ? do_page_fault+0x159/0x350
Jul 31 07:42:11 web1 kernel: [1673445.611846]  [<ffffffff812542d9>] sys_semctl+0x69/0xa0
Jul 31 07:42:11 web1 kernel: [1673445.611873]  [<ffffffff8100a0f2>] system_call_fastpath+0x16/0x1b
Jul 31 07:42:11 web1 kernel: [1673445.612094]  RSP <ffff8805a43f3dc8>
Jul 31 07:42:11 web1 kernel: [1673445.612322] ---[ end trace 11e0bd4d553eab55 ]---

Anybody know what this might indicate? Faulty memory/CPU? This is a first for me.

The process was listed with state "Ds", and didn't respond to either an apache stop command, kill -9 or kill -11. "reboot" and "reboot -f" also didn't go through. The latter only kicked me out of ssh, but didn't reboot the server. I had to do a physical reset via the hoster's admin console to get the server back to being responsive.

share|improve this question
1  
Are the pages served from local disk or over NFS mount? –  Janne Pikkarainen Aug 1 '11 at 7:57
    
Was this a one-time thing, or does it happen often? If it happens often, you might want to ask your hoster to check the machine's memory. –  Sgaduuw Aug 1 '11 at 9:52
    
So far, this was a one-time thing. Pages are served from local disk, with a webapp cache directory being served from a ramdisk (tmpfs). –  Eddy B. Aug 1 '11 at 11:10
    
was the webapp cache mounted and ok? –  Matt Simmons Aug 2 '11 at 3:51
    
As far as I know, yes, it was. It's something I'll double-check if this ever happens again. I'm not quite sure what could make it unmount, though, since that should only ever happen on a shutdown/reboot? Are there issues with tmpfs mounts disappearing for some reason? (and even if it unmounted, the app should simply continue using the hard disk directory that the tmpfs was mounted to, making the whole app run slower but not breaking apache?). –  Eddy B. Aug 4 '11 at 7:51
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.