are following assertions about linux load and tomcat right?

Question

I am looking to understand precisely how to use the load average and cpu usage on a Red hat machine that hosts a Tomcat 8 only. After looking on the net, I concluded the following assertions. Are the assertions right ? I am deeply sure of the first one since it comes from official Tomcat documentation. And I am confused about which processes could be in uninterruptible sleep.

1) Tomcat uses a thread to process a request, the maximum number of used threads is defined by Tomcat configuration ( see Tomcat documentation )

2) Oracle JVM works with native threads only since JRE 1.3 ( See JVM and threads I did not find an Oracle reference for this point)

3) Linux's run queue contains processes and threads (id native threads) the same way ( See Linux Load Averages: Solving the Mystery and Wikipedia )

4) Load average provides the average number of process/threads in the run queue ( See Linux Journal )

5) On a Linux machine running a Tomcat only, the load average provides almost the average number of requests.

6) On Linux the load average count process/threads in state running, runnable, and uninterruptible sleep ( See Linux Journal )

7) Process/threads in uninterruptible sleep are waiting for disks I/O, non interruptible locks, network I/O ( See Redhat documenationt that includes network I/O and Linux Load Averages: Solving the Mystery that does not include network I/0 )

The point 7 is not coherent with the reference See Linux Journal that says that "In fact, it is precisely the CPU load that is measured, because load averages do not include any processes or threads waiting on I/O, networking, databases or anything else not demanding the CPU. ".

I understood that if a process reads swap it is in uninterruptible sleep but if it reads a file on internal disk, nfs folder or a SAN bay, is it in uninterruptible sleep ? Red hat documentation listed network, if a process requests a resource on the network is it in uninterruptible sleep ?

Answer 1

1) Yes: Tomcat uses a thread to process a request.

2) Yes: Oracle JVM and OpenJDK JVM work with native threads only since JRE 1.3

3) Yes: Linux's run queue contains processes and threads (id native threads) the same way

4) Yes: Load average provides the average number of process/threads in the run queue

5) N/A

6) Yes: On Linux the load average counts processes/threads in state running, runnable, and uninterruptible sleep

7) This needs rephrasing. You seem to know how swap space works. So the big point: I/O to normal files is usually done with the same mechanism (not similar - literally the same one). This is called mmap. When your application wants to write to a file abc.txt it just changes bytes at a predefined memory address. The memory page (4096 bytes) is marked dirty. Soon it becomes written by a background daemon to filesystem (and not to swap space). And when application just opens file for reading it gets the memory address with initially no memory page. When it actually accesses the memory there the kernel reads the pages of the file and make them appear there. This is how general disk cache works in Linux - it is very much paging, nothing more.

Now the network part. Linux has a couple of ways to mount filesystems that don't reside on local SAS/SATA/FC disks, but in fact communicate over the network (for example NFS or CIFS). So paging the a.txt in fact causes packets to fly over the network. Your process doesn't have a socket in this specific case, just a mmapped address in its own address space. Everything it does is moving bytes between memory locations.

Uninterruptible sleep is defined as waiting for that mechanism - whether the filesystem is mounted from disk or from network. Lets say we're reading and we're waiting for the next page - the process needs to exist and needs to own the memory, because the memory has to be ready to receive the bytes being read from the disk (think: DMA in progress). That's why it stays despite you kill -9 it.

And the normal network traffic done with sockets owned by a process is a standard sleep, this one not counted in the load average.

A process could also read disks "the old way" without mmap, this would be also standard sleep. It's quite rare.

Answer 2

"Precise" is not a useful thing to achieve with load average, in my opinion. It is an algorithm, but somewhat arbitrarily developed aggregation of what a slow system feels like. From sched/loadavg.c:

This file contains the magic bits required to compute the global loadavg figure. Its a silly number but people think its important.

In particular, that Linux Journal article is a bit misleading to talk about load average as a CPU load metric. Since the TASK_UNINTERRUPTIBLE addition circa 1993, off CPU states such as I/O or certain locks held. Which makes it more of a system load metric. (And this is specific to Linux.)

Gregg's post you referenced, Linux Load Averages: Solving the Mystery did not have a complete list of these because

Nowadays, in Linux 4.12, there are nearly 400 codepaths that set TASK_UNINTERRUPTIBLE, including some lock primitives.

Rather that a complete list, that shows how to measure kernel uninterruptible via an off-CPU flame graph. Which is useful to understand why your workload's load average might be high even if CPU utilization may be low.

---

Good performance monitoring profiles application user response time, for the complete request.

Find any correlation of response time with load average. It may be that 6 load on a 4 CPU system is a symptom of degraded performance of your workload. (I made this number up, look at your data.)

When load is high, confirm the application is responding slowly, then dig deeper into every resource of the system to find out why.