SysRq, task list and missing processes

Question

I've been troubleshooting a pretty difficult issue on Linux where a JVM appears to hang. As part of the investigation, I've been trying to capture the list of current tasks and their state using one of the "magic" sysrq commands:

echo t > /proc/sysrq-trigger

What is astonishing is if I do this on a healthy system, I get a full list of all processes, their state and current call stack. When I do it with an "unhealthy" system (i.e. one where the JVM has hung somehow), some processes are missing. For example, the parent process of my multithreaded Java application is not being reported at all. This makes it really hard to draw any conclusions since I can't get an accurate snapshot of what's going on.

Anybody know if there are conditions where a process may be excluded from that report?

Thanks!

Answer 1

Is the JVM hanging because it is attempting to write to a process that isn't there anymore? From looking at the linux-source-2.6.38 code the sysrq trigger just calls:

/linux-source-2.6.38/include/linux/sched.h
283 /*
284  * Only dump TASK_* tasks. (0 for all tasks)
285  */
286 extern void show_state_filter(unsigned long state_filter);
287 
288 static inline void show_state(void)
289 {
290         show_state_filter(0);
291 }

Which calls:

/linux-source-2.6.38/kernel/sched.c
5485 void show_state_filter(unsigned long state_filter)
5486 {
5487         struct task_struct *g, *p;
5488 
5489 #if BITS_PER_LONG == 32
5490         printk(KERN_INFO
5491                 "  task                PC stack   pid father\n");
5492 #else
5493         printk(KERN_INFO
5494                 "  task                        PC stack   pid father\n");
5495 #endif
5496         read_lock(&tasklist_lock);
5497         do_each_thread(g, p) {
5498                 /*
5499                  * reset the NMI-timeout, listing all files on a slow
5500                  * console might take alot of time:
5501                  */
5502                 touch_nmi_watchdog();
5503                 if (!state_filter || (p->state & state_filter))
5504                         sched_show_task(p);
5505         } while_each_thread(g, p);
5506 
5507         touch_all_softlockup_watchdogs();
5508 
5509 #ifdef CONFIG_SCHED_DEBUG
5510         sysrq_sched_debug_show();
5511 #endif
5512         read_unlock(&tasklist_lock);
5513         /*
5514          * Only show locks if all tasks are dumped:
5515          */
5516         if (!state_filter)
5517                 debug_show_all_locks();
5518 }

There aren't any filters there, so its printing everything which suggest the parent is gone. Have you run lsof or ps and verified that it is still running?