0

We are working with huge hadoop cluster.

We have 736 datanode machines and each node have 16 core X 2 thread.

On some machines we saw CPU load average (98-128 for 5 min).

After deeply investigation , we identify that :

no HW problem
no Disk proble
no network problem
no infra problem

What we still not checked is about tune the linux parameters and tune kernel parameters.

What are the parameters or any kernel parameters that can help the machines in order to get good CPU working with most CPU LOW load average.

http://linuxrealtime.org/index.php/Improving_the_Real-Time_Properties

2
  • 1
    With a cluster that large then you should have some idea of where you can improve performance for the software load you're running, you're well past the point of general tuning suggestions.
    – Bob Dole
    Apr 16, 2019 at 19:49
  • Is this actually a problem for end user performance? Real problems have symptoms like longer job execution time, reduced data volumes, or longer interactive response time. Apr 16, 2019 at 23:41

2 Answers 2

0

Performance tuning is a complex and large area. The hardware, kernelparameters, softwaresettings, setup architecture and even more factors are to be considered.

As you are running a large setup, it makes sense to go deeper in that area, therefore I suggest to start with reading the RHEL Performance Tuning Guide. This should provide a good guide into the topic, with various suggestions and ideas.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/performance_tuning_guide/index

Your question contains too little information and detail to provide a reasonable suggestion in a direction of kernel parameter tuning.

0

I think you are running hadoop job at the time of CPU spike if so, consumption of resources depend on job complexity of job and parameters defined. I hope you are running yarn on that too? yarn is resources intensive application. Please use resources queue to regulate the resources consumption.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .