How to diagnose what Java process is doing in kernel space

Question

We have a problem with one Spark (Java) application on our Hadoop cluster. When this workload gets scheduled on 3 machines in our cluster kernel space CPU usage gets a very high. Example recording from sar:

09:35:52        CPU     %user     %nice   %system   %iowait    %steal     %idle
[...]
09:36:28        all     56.62      0.00     33.89      0.00      0.00      9.50
09:36:29        all     55.84      0.00     32.67      0.00      0.00     11.49
09:36:31        all     53.18      0.00     34.67      0.10      0.00     12.05
09:36:32        all     55.86      0.00     32.37      0.00      0.00     11.77
09:36:34        all     56.67      0.00     32.43      0.00      0.00     10.90
09:36:36        all     53.45      0.00     35.03      0.11      0.00     11.41
09:36:37        all     60.71      0.00     25.82      0.00      0.00     13.47
09:36:39        all     56.91      0.00     26.84      0.00      0.00     16.25

So far me managed to rule out:

• cgroup congestion - happened to us before
• Thermal issues - during the "event" we do not see drop of CPU frequencies.
• Does not look like something is waiting for IO, since iowait is low.
• Insufficient power to the machine - in this case we should see a drop in frequency and definitely not high kernel space CPU.

I have tried to run perf and look at flamegraphs, bud did not find anything interesting (apart from "it is java"). However I am inexperienced in this, so maybe I missed something on the graph: Perf flame graph

During the events we do not see anything different in dmesg than regular. We asked HP to check the hardware of these machines and they claim all is fine. We asked RedHat and our internal Linux team to investigate and they basically said "this is your application causing problems, please stop bothering us". The problem manifests only on 3 particular nodes in the whole cluster (when load lands up on other nodes it runs fine).

We have run out of ideas how to check what is going on. Could you please help us?

Our configuration is as follows:

• Cloudera CDH 6.1.1

• We have 10 machines in the cluster (plus 4 admin nodes) ONLY 3 have problems

• OS: RHEL 6.10 Santiago 2.6.32-754.25.1.el6.x86_64

• Machines: HP ProLiant SL4540 Gen8 (3 machines in one 4U chassis, problematic once are in the same box, but HP claims that HW is not the problem).

Answer 1

We have run out of ideas how to check what is going on. Could you please help us?

Interacting with kernel requires system calls, so try attaching strace(1) to your cluster process and see what's going on.

Probably start with something like

$ strace -cp $(cat /path/to/pid_file.pid)

to show your process' syscall summary, and start tracking from there.

Answer 2

First, what's the observed performance difference between the unhealthy and the healthy machines? Is there a factor of 2x or more? Do you even know?

What utilization do you see on other nodes?

"this is your application causing problems, please stop bothering us"

Do you have an evidence that it's your application that is causing the issue? You can examine pidstat output to see what proportion comes from "java" vs something else.

I'd then check number of context switched on the affected nodes and compare it to healthy ones. You can use perf stat -e cs -a -I 1000 for that.

While it might not be 100% applicable to your case, there's a nice example of an issue with lots of context switches in Brendan Gregg's LISA2019 Linux Systems Performance talk - first 8 minutes or so.

As @mforsetti suggested, you can also use strace but be very careful with running it on production servers - it can slow down your app by a factor of 100x (http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-syscall.html). As a limited strace alternative, it's possible to use perf trace.