we have Hadoop cluster and we are collection metrics collection data in order to investigate slowness behavior on spark applications

after long investigation on our Hadoop cluster

we noticed from Prometheus metrics point that node_disk_io_now is with high values more then normal , and its relevant for all HDFS disks on data-node machines

the node_disk_io_now definition is:

node_disk_io_now (field 9) The only field that should go to zero. Incremented as requests are given to appropriate struct request_queue and decremented as they finish.

we want to know , if tuning kernel parameters can gives positive aspects on disks performance

according to node_disk_io_now definition , seems that too many tasks are waiting in queue ,

and maybe some kernel parameters can help to improve the above bhavior so tasks in queue , will not be there for a long time

2 Answers 2


In your question I cant see what type of disk your are using, either SSD/SAS/NLSAS or hybrid. This is the most important part of your selection.

To Answer your question, for kernel parameters, I would say yes there are some parameters that you can play with, to find the best fit to your cluster.

The important part of IO optimization is the IO Scheduler. In current Linux Kernels there are three IO Schedulers.

NOOP(FIFO): Used for SAN,SSD type of storages where the storage device by itself is smart enough to take care of its own optimization.

DEADLINE: In general good for file and database servers. You have to consider whether you are enhancing either the throughput or latency. Or you can increase Read requests or vice versa.

CFQ: Not recommended for servers, good for Desktop usage.

Beside IO Schedulers there some other parameters for virtual memory which I will only mentioned them as below, but you need to read more about them yourself, to increase the performance.

  • 1
    about the DEADLINE - you mean about this settings ? cat /sys/block/device/queue/scheduler
    – King David
    Jun 16, 2022 at 6:39
  • @KingDavid Correct. Please bare in mind, to play with the items to find the best fit to your cluster. sometimes changing paraments have negative result. Jun 16, 2022 at 17:30

There is no (to my knowledge) Kernel options that can improve that. Howerver, you might want to check that all of your other nodes RAID controller have the same values (Block Write size for example) as that node, or the other way around actually.

Also make sure all your mounts in /etc/fstab have the same options.

Other than that, you need to check for your disks specs and make sure they are the same.

You could also check your RAM size as some file systems do heavy caching in RAM. Not sure about HDFS, but for example, ZFS rely a lot on that.

  • about memory on the serer we have 256G per server but the available memory is around - 30-40G do you think that the low available memory is part of the problem? ,
    – King David
    Jun 16, 2022 at 6:29
  • I found this doc: projectpro.io/article/… In the memory section, there is a kernel parameter to change that is related to the RAM. It might help you, but I doubpt. Look's more like a suggestion. Also section 2 might be interresting. The only problem I see in that is that your other nodes don't have that issue and don't have theses configs... Unless you did not config them ans you are not aware of theses changes. Other articles I found seem to be alligned with that one I shared.
    – yield
    Jun 16, 2022 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.