We have a problem with one Spark (Java) application on our Hadoop cluster. When this workload gets scheduled on 3 machines in our cluster kernel space CPU usage gets a very high. Example recording from sar:
09:35:52 CPU %user %nice %system %iowait %steal %idle [...] 09:36:28 all 56.62 0.00 33.89 0.00 0.00 9.50 09:36:29 all 55.84 0.00 32.67 0.00 0.00 11.49 09:36:31 all 53.18 0.00 34.67 0.10 0.00 12.05 09:36:32 all 55.86 0.00 32.37 0.00 0.00 11.77 09:36:34 all 56.67 0.00 32.43 0.00 0.00 10.90 09:36:36 all 53.45 0.00 35.03 0.11 0.00 11.41 09:36:37 all 60.71 0.00 25.82 0.00 0.00 13.47 09:36:39 all 56.91 0.00 26.84 0.00 0.00 16.25
So far me managed to rule out:
- cgroup congestion - happened to us before
- Thermal issues - during the "event" we do not see drop of CPU frequencies.
- Does not look like something is waiting for IO, since iowait is low.
- Insufficient power to the machine - in this case we should see a drop in frequency and definitely not high kernel space CPU.
I have tried to run perf and look at flamegraphs, bud did not find anything interesting (apart from "it is java"). However I am inexperienced in this, so maybe I missed something on the graph: Perf flame graph
During the events we do not see anything different in dmesg than regular. We asked HP to check the hardware of these machines and they claim all is fine. We asked RedHat and our internal Linux team to investigate and they basically said "this is your application causing problems, please stop bothering us". The problem manifests only on 3 particular nodes in the whole cluster (when load lands up on other nodes it runs fine).
We have run out of ideas how to check what is going on. Could you please help us?
Our configuration is as follows:
Cloudera CDH 6.1.1
We have 10 machines in the cluster (plus 4 admin nodes) ONLY 3 have problems
OS: RHEL 6.10 Santiago 2.6.32-754.25.1.el6.x86_64
Machines: HP ProLiant SL4540 Gen8 (3 machines in one 4U chassis, problematic once are in the same box, but HP claims that HW is not the problem).