The type of strace switches that I tend to use is this.
strace -ffttT -p pid -o /tmp/strace.out
An example of this would look like,
19:35:57.485493 mprotect(0x7f35e7472000, 16384, PROT_READ) = 0 <0.000037>
19:35:57.485599 mprotect(0x7f35e7692000, 4096, PROT_READ) = 0 <0.000030>
19:35:57.485697 mprotect(0x7f35e78b7000, 4096, PROT_READ) = 0 <0.000030>
19:35:57.485782 munmap(0x7f35e7896000, 129588) = 0 <0.000037>
19:35:57.485875 set_tid_address(0x7f35e78949d0) = 10730 <0.000029>
19:35:57.485960 set_robust_list(0x7f35e78949e0, 0x18) = 0 <0.000024>
19:35:57.486048 futex(0x7fff8f58628c, FUTEX_WAKE_PRIVATE, 1) = 0 <0.000025>
19:35:57.486131 futex(0x7fff8f58628c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 7f35e7894700) = -1 EAGAIN (Resource temporarily unavailable) <0.000024>
You see the time difference at the right hand side of the system call showing how much time it has taken to go from one system call to another.
It will catch you the time difference between the system calls. So, when you see that a system call has quite a few seconds gap with the next system call, then there it is making some noise.
Another method is to coredump it with gcore. However, that requires a little experience navigating through gdb.
But, if the thread is a kernel thread, then you can't strace or coredump it. In that case, we have to use something more complex. In RHEL5 kernel, we use oprofile. In RHEL6, we use perf. I prefer perf over oprofile. Perf data can be collected with graph like format showing the system call where maximum percentage of CPU is being used.
With a test perf, I see this.
38.06% swapper [kernel.kallsyms] [k] mwait_idle_with_hints ↑
29.45% swapper [kernel.kallsyms] [k] read_hpet
4.90% swapper [kernel.kallsyms] [k] acpi_os_read_port ▒
4.74% swapper [kernel.kallsyms] [k] hpet_next_event
It shows the kernel function where 38% CPU time is being spent. Now, we can check the function and see what it is doing and what it is supposed to do.
With a few examples, it is not that much hard.