we have Hadoop cluster and we are collection metrics collection data in order to investigate slowness behavior on spark applications
after long investigation on our Hadoop cluster
we noticed from Prometheus metrics point that node_disk_io_now is with high values more then normal , and its relevant for all HDFS disks on data-node machines
the node_disk_io_now definition is:
node_disk_io_now (field 9) The only field that should go to zero. Incremented as requests are given to appropriate struct request_queue and decremented as they finish.
we want to know , if tuning kernel parameters can gives positive aspects on disks performance
according to node_disk_io_now definition , seems that too many tasks are waiting in queue ,
and maybe some kernel parameters can help to improve the above bhavior so tasks in queue , will not be there for a long time