I was having the common issue of not being able to restart httpd because of a high number of locked semaphores. After clearing those and restarting httpd, I also doubled the semaphores.
However, I'm lead to believe some Apache process has a memory leak and it's not releasing them, so I'll still run into the issue but later.
------ Semaphore Arrays -------- key semid owner perms nsems 0x00000000 13467648 apache 600 1 0x00000000 13697025 apache 600 1 0x00000000 13729794 apache 600 1 0x00000000 13762563 apache 600 1 0x00000000 13795332 apache 600 1 0x00000000 14057477 apache 600 1 0x00000000 14123014 apache 600 1 0x00000000 14155783 apache 600 1 0x00000000 14188552 apache 600 1 0x00000000 14221321 apache 600 1 0x00000000 14254090 apache 600 1
So let's trace down what process was managing those
ipcs -s -i 13697025:
Semaphore Array semid=13697025 uid=48 gid=48 cuid=0 cgid=0 mode=0600, access_perms=0600 nsems = 1 otime = Not set ctime = Thu Jul 11 03:41:01 2019 semnum value ncount zcount pid 0 1 0 0 18395
And finally what corresponds to the pid
ps --pid 18395:
PID TTY TIME CMD
So am I reading that right the second semaphore in the list belongs to a process that already died and didn't clean up?
For example, running through the process with the second to last in the list produced that it actually did belong to a running process:
PID TTY TIME CMD 22331 ? 00:03:42 httpd
It's obvious these are owned by Apache, but what would be the best way of debugging what's causing this. We have in the range of 50+ virtual hosts is there a way to log an actual server request and trace it back to the process spawned, etc?
I have reason to believe it's the
php-sqlsrv.x86_64 extension or something related to it because of the time relation between when that was configured and when the issue started happening. I have the ability to roll back but just want to learn how to go deep into debugging, perhaps even to submit a patch for the bug.