5

I'm running a CentOS server as a LAMP stack serving a custom php application. At seemingly randomly intervals it will slow down. Looking at the server-status page I see the PID list locked up with the several of the same ajax call all requested from one user's client IP. (The IP changes but it is always just one)

I see the 'M' parameters status is W for "Sending Reply", what could this mean?

The slowness usually resolves on it's own after 5 minutes to an hour. However the other day I decided to perform a:

service httpd restart graceful.

This totally resolved the problem - for 10 minutes. Below is the server-status 14 minutes later, slow and locked up. It seems the requests quickly build up to 50 and the server slows down.

Points to consider:

  • Multiple requests always from same IP
  • Requests last a maximum time (SS) of ~ 200 seconds
  • All requests going to one ajax.php script
  • Slowdowns sometimes don't happen for a few weeks, then several within a few days
  • Users only have a couple of tabs open in browser of server address ~ 25 total
  • Apparently worst problems occur late afternoon

enter image description here

So my question is : What could be causing this lock up, and why are all the requests "Sending Reply"?

Here is the httpd.conf

<IfModule worker.c>
StartServers         2
MaxClients          50
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>

2 Answers 2

4

Looks like those connections get stuck for a long while (SS is the time the request has taken, and some of those are pushing several minutes).

My instinct tells me to look at the database and the PHP application. Check so that you have enough available connections in the pool, check for maintenance tasks (full vaccum can lock the database for a long period of time!) and log long queries to see if you are doing something that may lock an important table. It can also be a problem in the PHP script that prevents it from terminating in a timely manner.

Here is a page with some useful debugging tips for this kind of situations.

0

Given that in your case the connections are from your LAN, it's unlikely to be an attack, but in my case I had 1 external IP (at a time) doing the same things on a wordpress site (latest) and get this, even on stuff like:

/wp-content/plugins/wpmarketplace/readme.txt

which doesn't exist on my server (most of the GET resources didn't exist and there were plenty of txt and css files being GETed). There were also POST requests on various php files, leading to the same slowness and eventually freeze.

So my hunch is, this is a very poorly written script to check for vulnerable sites, resulting in a DoS. Or it could actually be a DoS and not a buggy script, however it's been ages since I've seen one, nowadays people do DDoSes.

I'm currently working on some scripts to get this under control. Once I have them, I'll post back, maybe it will help someone.

Laster edit:

after much testing, I think I finally managed to get things under control. Let's assume you make a new script /root/check_httpd.sh (explanations at the bottom)

cnt=`ps -Af | grep httpd | grep -v rotatelogs | grep -v grep | wc -l`
now=`date +%Y-%m-%d_%H-%M`

# change the 40 below to something meaningful to your server
if [ $cnt -ge 40 ]
then
  /usr/bin/wget -q -O /root/apache_status_$now http://<your server here>/server-status
  /sbin/service httpd restart
fi

# change hda to your partition/disk which is being "killed" by httpd during the freeze
dsk=`/usr/bin/iostat -dx /dev/hda 5 2 | grep hda | tail -1 | awk '{print $12}'`

if (( $(echo "$dsk > 98" |bc -l) ))
then
  /bin/sleep 5
  dsk=`/usr/bin/iostat -dx /dev/hda 5 2 | grep hda | tail -1 | awk '{print $12}'`
  if (( $(echo "$dsk > 98" |bc -l) ))
  then
    /sbin/service httpd restart
  fi
fi

You then add this to cron like:

0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58 * * * * /root/check_httpd.sh

don't forget to

chmod +x /root/check_httpd.sh

And the explanations.

So in my case, initially, I noticed that (during freeze) the httpd status page would show a lot of "W" state httpd children, with various wait times on various resources, some valid, some invalid. I spent a lot of time with various options to get a 90+% scenario based on the status page to find when the server is frozen and not under heavy usage. No luck. But then I figured that normally, even under "heavy" load, my httpd child process count would still be under 20-30 (my site is "lite"), so I made a few tests and found that a count of 40 httpd child count always happens during freeze (NOTE: you can remove the status wget from that section, it is there for you to confirm that during the restart for whatever value count you choose, there is indeed a freeze. You manually check that)

However, this alone would not cut it. I still had cases when the server froze up for over 24 hours before that 40 count would kick in. Searching some more I found atop utility which i left running in a putty terminal, so whenever the server froze I could see what exact resources were being consumed that much. And I noticed it was the HDD. So there came the second check for the HDD usage, But as you know, hdd usage spikes every now and then, so 1 check alone results in false positives. What I did was to make another check after a few seconds wait and only then reboot the httpd if needed.

You will need to play a bit on your server and fine tune the threshold values to make them fit for your environment and usage patterns.

1
  • 1
    That crontab entry can be rewritten as */2 * * * * /root/check_httpd.sh (runs on every even minute) Apr 27, 2020 at 7:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .