I mention Apache reload in the title since this is the only lead I've got, but to be honest I doubt it is related to Apache at all.
I was reloading Apache today after using tcpdump to diagnose a website that wasn't sending email on an Ubuntu 10.10 box running inside a VMware EX 4.1 64 bit virtual machine. My ssh session completely froze, websites were unavailable, and I had to log in to the VMware console and pull a 'Raising Elephants ...' to revive the system. On the (frozen) console was the following repeating message, every 5 seconds or so:
[...] Call Trace:
[...] [<...>] ? do_sync_read+0xda/0x120
[...] [<...>] ? sock_destroy_inode+0x35/0x40
[...] [<...>] ? __d_free+0x3f/0x60
[...] [<...>] ? d_free+0x58/0x60
[...] [<...>] ? apparmor_file_permission+0x18/0x20
[...] [<...>] ? security_file_permission+0x16/0x20
[...] [<...>] ? sys_semop+0x10/0x20
[...] [<...>] ? system_call_fastpath+0x10/0x1b
[...] Code: ...
I got this snippet from /var/log/kern.log although this might be normal since I used the Alt + SysReq + REISUB key combo to restart the server:
Aug 4 22:24:18 light-app04 kernel: [ 3.134642] EXT4-fs (dm-1): INFO: recovery required on readonly filesystem
Aug 4 22:24:18 light-app04 kernel: [ 3.134647] EXT4-fs (dm-1): write access will be enabled during recovery
Aug 4 22:24:18 light-app04 kernel: [ 8.399828] EXT4-fs (dm-1): orphan cleanup on readonly fs
Aug 4 22:24:18 light-app04 kernel: [ 8.399840] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3659
Aug 4 22:24:18 light-app04 kernel: [ 8.399960] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3606
Aug 4 22:24:18 light-app04 kernel: [ 8.399971] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3614
Aug 4 22:24:18 light-app04 kernel: [ 8.399980] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3596
Aug 4 22:24:18 light-app04 kernel: [ 8.399989] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3589
Aug 4 22:24:18 light-app04 kernel: [ 8.399999] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3586
Aug 4 22:24:18 light-app04 kernel: [ 8.416270] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3584
Aug 4 22:24:18 light-app04 kernel: [ 8.416289] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3583
Aug 4 22:24:18 light-app04 kernel: [ 8.416304] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 3442
Aug 4 22:24:18 light-app04 kernel: [ 8.416316] EXT4-fs (dm-1): ext4_orphan_cleanup: deleting unreferenced inode 2810
Aug 4 22:24:18 light-app04 kernel: [ 8.416325] EXT4-fs (dm-1): 10 orphan inodes deleted
Aug 4 22:24:18 light-app04 kernel: [ 8.416328] EXT4-fs (dm-1): recovery complete
Aug 4 22:24:18 light-app04 kernel: [ 8.417595] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (nul
Every log file just stops logging at the time of the crash and restarts after I rebooted the system.
I had this exact same problem and message once before, also when I was using the terminal. It never happened during normal operation.
We run munin on this server, and I see odd thin spikes just before the server crashed on the following graphs. Keep in mind that I was using tcpdump (to the console, not disk) a few minutes before and munin's resolution is 5 minutes. A lot could be attributed to that:
- Disk IO: from almost none to 25 writes per second
- Disk throughput: from 5K to 100K per second written
- IOStat: from 20 to 250 blocks per second written
- Eth0 traffic: from 0 to 2.5 mbps incoming.
- CPU: just 2-4%
- Memory: never over 25%
- Interrupts: from 100 to 1200 per second (on eth0 mostly)
I can't place the call trace at all, and the amount of incoming network and disk activity strikes me as odd as I was only working on sending email. Other websites reside on this server as well, but I'd expect a spike in Apache throughput causing the incoming traffic, which wasn't the case.
Any ideas on the possible cause? Thanks!
