Asterisk using 100% of the cpu and freezing my debian host

Question

I have a Debian host

$ uname -a

Linux voip 2.6.25-2-amd64 #1 SMP Mon Jul 14 11:05:23 UTC 2008 x86_64 GNU/Linux

Old asterisk

asterisk 1:1.4.21.2~dfsg-3

It has been working for a long-long time

$uptime

13:50:37 up 1047 days, 21:02, 7 users, load average: 0.17, 0.19, 2.81

Today the host was froze, pings were answered , but I could not ssh, no dhcp or any other services.

After I logged in from the console and looked into top -I saw asterisk was using 100% of the cpu. killall asterisk didn't work, I had to resort to killall -9. After I restarted the service, everything seemed fine.

I checked my logs and noticed this:

> Mar 12 13:18:06 voip kernel: [96834831.947297] asterisk      D
> ffff81000100a080     0 15757      1                 Mar 12 13:18:06
> voip kernel: [96834831.947304]  ffff8100390fdda8 0000000000000086
> ffff810039561c00 ffff810024af33d8                                     
> Mar 12 13:18:06 voip kernel: [96834831.947310]  ffff81002dda8080
> ffff810039cb2d40 ffff81002dda82f0 0000000000000001                    
> Mar 12 13:18:06 voip kernel: [96834831.947317]  0000000000000296
> 0000000000000003 ffff8100390fdda8 ffffffff8022bea4                    
> Mar 12 13:18:06 voip kernel: [96834831.947322] Call Trace:            
> Mar 12 13:18:06 voip kernel: [96834831.947345]  [<ffffffff8022bea4>]
> __wake_up+0x38/0x4e                           Mar 12 13:18:06 voip kernel: [96834831.947364]  [<ffffffff881145b4>]
> :jbd:log_wait_commit+0x9f/0xed                Mar 12 13:18:06 voip
> kernel: [96834831.947373]  [<ffffffff80246a54>]
> autoremove_wake_function+0x0/0x2e             Mar 12 13:18:06 voip
> kernel: [96834831.947389]  [<ffffffff881104f7>]
> :jbd:journal_stop+0x165/0x18d                 Mar 12 13:18:06 voip
> kernel: [96834831.947400]  [<ffffffff802b4c6d>]
> __writeback_single_inode+0x17f/0x29d          Mar 12 13:18:06 voip kernel: [96834831.947407]  [<ffffffff80246a54>]
> autoremove_wake_function+0x0/0x2e             Mar 12 13:18:06 voip
> kernel: [96834831.947419]  [<ffffffff802b558b>] sync_inode+0x24/0x31  
> Mar 12 13:18:06 voip kernel: [96834831.947440]  [<ffffffff881213ce>]
> :ext3:ext3_sync_file+0x8a/0x9c                Mar 12 13:18:06 voip
> kernel: [96834831.947449]  [<ffffffff802b79c2>] do_fsync+0x52/0xa4    
> Mar 12 13:18:06 voip kernel: [96834831.947456]  [<ffffffff802b7a37>]
> __do_fsync+0x23/0x36                          Mar 12 13:18:06 voip kernel: [96834831.947463]  [<ffffffff8020bd9a>]
> system_call_after_swapgs+0x8a/0x8f

I am discombobulated, is it an Asterisk bug? Or something else? Why do I see do_fsync and sync_inode in the log? Perchance it's a filesystem or hard-disk problem? (I use lvm+ext3)

Can anyone point me in the right direction?

Answer 1

If something has been working about three years in a row without a hitch and suddenly starts to throw kernel errors related to file system, I would say your hardware is starting to melt. Immediately make sure you have usable backups and replace the server immediately with a new one.

Or, at minimum, replace the hard drives, even though problems that look like a HD problem can sometimes be because of bad RAM/CPU or overheating.

Answer 2

I think you might have a hardware problem. Try running fsck to repair your filesystem.