RHEL + can we improve disks performance by tuning kernel parameters?

Question

we have Hadoop cluster and we are collection metrics collection data in order to investigate slowness behavior on spark applications

after long investigation on our Hadoop cluster

we noticed from Prometheus metrics point that node_disk_io_now is with high values more then normal , and its relevant for all HDFS disks on data-node machines

the node_disk_io_now definition is:

node_disk_io_now (field 9) The only field that should go to zero. Incremented as requests are given to appropriate struct request_queue and decremented as they finish.

we want to know , if tuning kernel parameters can gives positive aspects on disks performance

according to node_disk_io_now definition , seems that too many tasks are waiting in queue ,

and maybe some kernel parameters can help to improve the above bhavior so tasks in queue , will not be there for a long time

Answer 1

In your question I cant see what type of disk your are using, either SSD/SAS/NLSAS or hybrid. This is the most important part of your selection.

To Answer your question, for kernel parameters, I would say yes there are some parameters that you can play with, to find the best fit to your cluster.

The important part of IO optimization is the IO Scheduler. In current Linux Kernels there are three IO Schedulers.

NOOP(FIFO): Used for SAN,SSD type of storages where the storage device by itself is smart enough to take care of its own optimization.

DEADLINE: In general good for file and database servers. You have to consider whether you are enhancing either the throughput or latency. Or you can increase Read requests or vice versa.

CFQ: Not recommended for servers, good for Desktop usage.

Beside IO Schedulers there some other parameters for virtual memory which I will only mentioned them as below, but you need to read more about them yourself, to increase the performance.

vm.swappiness
vm.dirty_ratio
vm.dirty_background_ratio
vm.dirty_expire_centisecs
and....

Answer 2

There is no (to my knowledge) Kernel options that can improve that. Howerver, you might want to check that all of your other nodes RAID controller have the same values (Block Write size for example) as that node, or the other way around actually.

Also make sure all your mounts in /etc/fstab have the same options.

Other than that, you need to check for your disks specs and make sure they are the same.

You could also check your RAM size as some file systems do heavy caching in RAM. Not sure about HDFS, but for example, ZFS rely a lot on that.