I have a debian VM I'm running on an ESX4.0 server. This VM hosts a number of users, each running an irssi session inside a screen instance.
This is working quite well, except for one user. For some reason, this irssi session keeps peaking at 100% CPU usage (while continuing to work normally). It's not running any scripts that aren't loaded on other irssi sessions that are behaving more sanely.
The 100% cpu doesn't start immediately, but usually within a few hours after starting up. It never goes away.
How would you go about debugging the source of this problem? I tried using strace on it, and didn't see anything immediately obvious, though there is certainly a different pattern to the calls at start, and after it peaks.
At start, here's the histogram of calls over 30 seconds:
time: 334 gettimeofday: 317 poll: 122 read: 9 write: 2 restart_syscall: 1
Once the CPU starts pegging, I get this:
gettimeofday: 230176 read: 115122 poll: 115106 time: 531 write: 107 waitpid: 38 _llseek: 2 ioctl: 2 fstat64: 2 open: 2 close: 2 fcntl64: 2 unlink: 1
A histogram of 'ltrace -S' of the pegging process shows these as the top entries:
SYS_read: 61731 g_io_channel_read: 34115 SYS_gettimeofday: 24662 SYS_poll: 12344 fflush: 6828 g_main_context_iteration: 6823 __ctype_toupper_loc: 4025 g_strcasecmp: 3757 g_hash_table_lookup_extended: 3325 g_direct_hash: 3068
What am I missing? What's the next step to solving this?