We have experienced a very strange problem. The load of the server machine is very high due to high disk read io. But the processes running on this server do not perform any disk read operation. We also noticed that when we execute top command, for most of processes, the values in "SHR" column are zero. Compared with other normal servers, we found that by executing "free -m", the result shows that the buff/cache value in this server is lower than the value of other normal servers. Swap is not enabled on this server. What could be the reason for this issue?

The centos version:CentOS Linux release 7.3.1611

kernel version:3.10.0-693.21.1.std7a.el7.0.x86_64

Here are screenshots of pidstat,free,vmstat command result

enter image description here enter image description here enter image description here

  • Have you tried iotop? It should tell you which processes generate I/O. Mar 5 at 2:56
  • @berndbausch I tried. But I don't think it is caused by disk io as the processes on this service do not perform io read.
    – yifan
    Mar 5 at 3:25
  • You say that disk IO is not caused by disk IO? This is puzzling. Mar 5 at 3:28
  • @berndbausch yes, quite weird. I think it has something to do with memory. maybe it is caused by memory swap in or swap out?
    – yifan
    Mar 5 at 3:32
  • @yifan how did you find that processes running on the machine aren't causing the high disk read IO? Low buff/cache might indicate that the system doesn't have enough memory to cache files contents in kernel memory ("page cache"). Mar 5 at 4:38

The data you provided will not tell anyone who is reading what and why. Even if your application code does no reading at all, the JVM, Libraries and Java bytecode need to be loaded into memory, for example, and the filesystem might be doing some (metadata) reads, or log rotation might involve compressing old logfiles, etc.

You may want to do some kind of profiling to get this kind of information, this could be as simple as making one or more (manual) Java stackdumps, see: https://www.baeldung.com/java-thread-dump or do this with some kind of profiler, maybe something that also keeps an eye on the native stack and can produce flamegraphs, like: http://www.brendangregg.com/offcpuanalysis.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.