To make a long story short, I have two identical servers hosting VMs using VirtualBox and both servers host one VM each nearly setup the same way, only that one is production and the other for internal tests and development. The important thing ist that the server's hardware and OS is identical, that both VMs use the same OS and run mostly the same software as well, only the usage scenario is a bit different. The problem is that I'm encountering massive performance issues on the production VM for some workloads after some runtime, which I'm unable to reproduce for the internal VM.
All the software in the VM is working "fine" in general, no errors, it's only that some workloads are able to put such a high load/overhead/whatever on the VM that it becomes extremely slow and unusable. After some hours of runtime, even a restart of the ClamAV-daemon is already triggering the problem. I can trigger it with some special load on Tomcat as well and all cases have massive CPU and at least some I/O in common. But only in prod, test-VM with same amount of CPUs, RAM etc. works as expected. Even in prod-VM the problem doesn't occur directly after a restart, seems to only be after some hours of runtime.
What I'm doing now is comparing sysctl -a of both systems and check which of the differences might lead to my performance problems. One difference is the following:
fs.aio-max-nr = 65536
fs.aio-nr = 0
vs.
fs.aio-max-nr = 65536
fs.aio-nr = 2661
The first is the production VM. I have some other VMs with 0 as well, but some with non-0, too. Because prod- and test-VM host very similar software, httpd, Tomcat7, Postgres 9.6, custom Perl services etc., it doesn't make any sense to me that one is having 0, while the other doesn't. From what I've read, 0 simply means that no one makes use of async I/O in the prod-VM, but in the test-VM. Which is very unlikely because of the same software in use.
So I'm guessing that there is some configuration difference for some reason which leads software in prod-VM to think that it can't make use of async I/O, which might degrade performance massively in my use case.
While aio-max-nr obviously is not the problem, are there other settings, packages, libs or whatever which might influence that software thinks async I/O is not available?
The only things I've found were software specific, but not related to the software I'm using, or mentioned fs.aio-max-nr as a possible bottleneck, which is not the case for me obviously.
In the past, the following to at least check if async I/O is available in principle seemed to work, which doesn't seem to be the case anymore, on none of my systems anything is found.
grep kio /proc/slabinfo
https://kbflow.wordpress.com/2013/02/25/check-if-async-io-is-enabled-in-centos/ https://www.systutorials.com/linux-kernels/125888/patch-aio-remove-kioctx-from-mm_struct-linux-2-6-15/
The following does provide some data and results the same on both systems:
ls -l /sys/kernel/slab | grep kio
lrwxrwxrwx 1 root root 0 Apr 18 13:03 aio_kiocb -> :t-0000128
lrwxrwxrwx 1 root root 0 Apr 18 13:02 kioctx -> :t-0000640
https://community.oracle.com/message/14732908#14732908
Not sure what that data tells me, though, some data is equal on both VMs, some differs, especially objects_partial is 0 again at the prod-VM. I've hoped to find some simple switch or such in some config file. :-)
AIO is enabled in the kernel itself:
cat /boot/config-4.4.0-119-generic | grep AIO
CONFIG_AIO=y
CONFIG_COMEDI_AIO_AIO12_8=m
CONFIG_COMEDI_AIO_IIRO_16=m
CONFIG_DELL_WMI_AIO=m
