server hang-up; no response, but no disconnections either

Question

Greetings Experts,

System: standard gentoo install with not much more than postfix.

I was in the middle of a big IMAP-driven email-migration last night (a bunch of Perl scripts) when the new mail server stopped responding. However, my SSH connections are still live and will not drop. New connections hang (before authentication), but do not time out.

Does this mean that it will eventually recover? Or need I reboot the server?

Answer 1

What does dmesg|tail says?

Answer 2

Perhaps the kernel did run out of entropy. That can happen at least with Cyrus IMAP server if it is not configured to use /dev/urandom as its randomness source and if the server is unable to generate enough "real" randomness to /dev/random. Your symptoms match the ones I encountered years ago.

To check if that is the case for you, let

watch -n1 'cat /proc/sys/kernel/random/entropy_avail' 

or

while true; do cat /proc/sys/kernel/random/entropy_avail >>/somepath/available_entropy.txt; sleep 1; done

run and see if the available entropy constantly drops to or near 0 during the IMAP hang. In case that happens, the IMAP server software is waiting for new randomness.

One way to get more entropy is to install rngd, in Gentoo's case that means emerge rng-tools and then starting up the rngd (Random Number Gathering Daemon). It shovels semi-randomness from /dev/urandom to /dev/random if the real randomness is running too low.

Obligatory warning: in über-secure environments this is not what you want.

Answer 3

Reboot was required. Waiting for it resulted in nothing happening.

Upon further investigation, it was discovered that there was a memory leak in the Perl IMAP library being used. Originally, I had set up the Perl script to load all email accounts into an array (from a text file referenced from arg 1 on command-line), then loop through them running the migration code for each account. For each iteration of the loop, the script logged-in to both the source mail server and target mail servers, ran the migration code, then logged-out of both mail servers. This eventually ate up all available RAM, then all available SWAP, until finally init killed the process.

I thought I would speed things up: I used screen to run 9 of these processes, each on a different set of accounts. After launching nine of these processes, the system quickly slowed to a crawl, then stopped responding all together. I'm guessing init would have eventually killed all the Perl processes, but how long would that haven taken? Thus, a reboot was required.

I modified my Perl migration script to do one account, then exit. Then I set up a bash loop like this to loop through all accounts sourced from the same text file:

# cat run_migration4.sh
#!/bin/bash

FILE=$1

# read $FILE using the file descriptors
exec 3<&0
exec 0<$FILE

while read line
do
        # use $line variable to process line
        echo line: $line
        ./migration4.pl $line
done
exec 0<&3

This worked extremely well. I was able to run nine of these in one screen session and they all took up an insignificant amount of RAM, no where near the 4G limit of the server. The server load average never got above 2 or 3. All completed without issue.