Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a system where some custom software is running, and then may receive a SIGHUP. As a result of the SIGHUP, the process does some cleanup and terminates, but it doesn't terminate its long running child processes.

After the system was running for several months (and in the course of that many of my processes had the misfortune of being hanged up on), I can see in my Ubuntu 12.04 server that I have many of the child processes that have init as their parent, but they are not running nor stopped or zombie - ps shows them in "sleep" state.

If I try to strace one of these processes, the process starts running then immediately sees that it trying to process files that are no longer there (temporary files that were created and deleted some months ago) and exits:

# strace -fp 16111
Process 16111 attached - interrupt to quit
open("/tmp/some-file", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
write(2, "/tmp/some-file:"..., 59) = 59
close(3)                                = 0
exit_group(1)                           = ?
Process 16111 detached

This also happens if I try to kill -STOP <pid> (at least I guess it does - after I send the signal, the process disappears).

Any idea what's going on?

share|improve this question
    
Looks like the process is blocked on IO. So if the file gets missing, only timeout or signal can unblock it. But it depends a lot on the code. This is just a supposition. –  aif Apr 27 at 12:36
    
While your supposition kind of makes sense because the parent process has a FIFO that is read by the child process, which obviously closes when the parent is hanged up, but if the process is blocked on IO, I would expect it status to be D not S. –  Guss Apr 27 at 19:29

1 Answer 1

up vote 0 down vote accepted

The process may have received one of the terminal stop signals. Those may be send if a background process tries to communicate with the terminal. There are three types of terminal stop signals: TSTP, TTIN, and TTOU. My first guess is the process tried to read from the terminal while it was backgrounded, and thus got send a TTIN signal. By the time you resume it by attaching strace, there no longer is a terminal, so it gets an error code instead of a signal, if the read is retried. Another way to unfreeze the process is by sending it a CONT signal.

share|improve this answer
    
The attachment to a terminal makes sense as the main process was started from an SSH command (that was disconnected after backgrounding the process), but from my testing all the signals mentioned above cause the process to get into a T state, while the processes I have problem with are in the S state. Also, sending them CONT has no effect. –  Guss Apr 27 at 20:04
1  
The ptrace implementation in the kernel makes use of signal internally in a way, which unfortunately sometimes affects the behavior of programs. The first line or two of the strace output might provide a hint. The wchan column of ps output while the process is sleeping might provide a hint as well. ps -o user,pid,ppid,lstart,tname,wchan,cmd may be a useful selection of columns to look at in your case. –  kasperd Apr 27 at 20:12
    
All the relevant processes have either pipe_w under "wchan", or wait (for shell processes waiting on the blocked processes). What does it mean? –  Guss Apr 30 at 7:45
    
pipe_w sounds like it may be blocked trying to write to a pipe. That would indicate that the process at the reading end of that pipe is not actually reading from it. I can't say why this would change from using strace, which is why I asked for the first two lines of the strace output. I have seen a similar problem happen when starting a program with ssh -fX where the program was also writing to stdout or stderr. It would write until the pipe buffer was full and then the window would freeze. Through /proc I was able to find the read end owned by sshd and read from it to clear the buffer. –  kasperd May 1 at 14:29
1  
I would check that one more time, because it is not supposed to be possible to be blocking on a pipe, if the other end of said pipe has been closed. –  kasperd May 2 at 7:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.