Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I mention Apache reload in the title since this is the only lead I've got, but to be honest I doubt it is related to Apache at all.

I was reloading Apache today after using tcpdump to diagnose a website that wasn't sending email on an Ubuntu 10.10 box running inside a VMware EX 4.1 64 bit virtual machine. My ssh session completely froze, websites were unavailable, and I had to log in to the VMware console and pull a 'Raising Elephants ...' to revive the system. On the (frozen) console was the following repeating message, every 5 seconds or so:

[...] Call Trace:
[...]   [<...>] ? do_sync_read+0xda/0x120
[...]   [<...>] ? sock_destroy_inode+0x35/0x40
[...]   [<...>] ? __d_free+0x3f/0x60
[...]   [<...>] ? d_free+0x58/0x60
[...]   [<...>] ? apparmor_file_permission+0x18/0x20
[...]   [<...>] ? security_file_permission+0x16/0x20
[...]   [<...>] ? sys_semop+0x10/0x20
[...]   [<...>] ? system_call_fastpath+0x10/0x1b
[...] Code: ...

I got this snippet from /var/log/kern.log although this might be normal since I used the Alt + SysReq + REISUB key combo to restart the server:

Aug  4 22:24:18 light-app04 kernel: [    3.134642] EXT4-fs (dm-1): INFO: recovery required on readonly filesystem
Aug  4 22:24:18 light-app04 kernel: [    3.134647] EXT4-fs (dm-1): write access will be enabled during recovery
Aug  4 22:24:18 light-app04 kernel: [    8.399828] EXT4-fs (dm-1): orphan cleanup on readonly fs
Aug  4 22:24:18 light-app04 kernel: [    8.399840] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3659
Aug  4 22:24:18 light-app04 kernel: [    8.399960] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3606
Aug  4 22:24:18 light-app04 kernel: [    8.399971] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3614
Aug  4 22:24:18 light-app04 kernel: [    8.399980] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3596
Aug  4 22:24:18 light-app04 kernel: [    8.399989] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3589
Aug  4 22:24:18 light-app04 kernel: [    8.399999] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3586
Aug  4 22:24:18 light-app04 kernel: [    8.416270] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3584
Aug  4 22:24:18 light-app04 kernel: [    8.416289] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3583
Aug  4 22:24:18 light-app04 kernel: [    8.416304] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3442
Aug  4 22:24:18 light-app04 kernel: [    8.416316] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 2810
Aug  4 22:24:18 light-app04 kernel: [    8.416325] EXT4-fs (dm-1): 10 orphan inodes deleted
Aug  4 22:24:18 light-app04 kernel: [    8.416328] EXT4-fs (dm-1): recovery complete
Aug  4 22:24:18 light-app04 kernel: [    8.417595] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (nul

Every log file just stops logging at the time of the crash and restarts after I rebooted the system.

I had this exact same problem and message once before, also when I was using the terminal. It never happened during normal operation.

We run munin on this server, and I see odd thin spikes just before the server crashed on the following graphs. Keep in mind that I was using tcpdump (to the console, not disk) a few minutes before and munin's resolution is 5 minutes. A lot could be attributed to that:

  • Disk IO: from almost none to 25 writes per second
  • Disk throughput: from 5K to 100K per second written
  • IOStat: from 20 to 250 blocks per second written
  • Eth0 traffic: from 0 to 2.5 mbps incoming.
  • CPU: just 2-4%
  • Memory: never over 25%
  • Interrupts: from 100 to 1200 per second (on eth0 mostly)

I can't place the call trace at all, and the amount of incoming network and disk activity strikes me as odd as I was only working on sending email. Other websites reside on this server as well, but I'd expect a spike in Apache throughput causing the incoming traffic, which wasn't the case.

Any ideas on the possible cause? Thanks!

share|improve this question
    
I wouldn't be too concerned at the disk spike - but such a spike in the inbound network traffic is odd. Have you tried putting wireshark on a mirror-port on your switch to try and capture the spike - might give you more clues... –  Mike Insch Aug 4 '11 at 21:26
    
No, but the spike disappeared after the reboot. And I have no clue what caused it so there is no way to reproduce it. Will have to anticipate the spike to be able to capture it.. –  Kamiel Wanrooij Aug 4 '11 at 21:29
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.