We are working with huge hadoop cluster.
We have 736 datanode machines and each node have 16 core X 2 thread.
On some machines we saw CPU load average (98-128 for 5 min).
After deeply investigation , we identify that :
no HW problem
no Disk proble
no network problem
no infra problem
What we still not checked is about tune the linux parameters and tune kernel parameters.
What are the parameters or any kernel parameters that can help the machines in order to get good CPU working with most CPU LOW load average.
http://linuxrealtime.org/index.php/Improving_the_Real-Time_Properties