0

We am running LAMP stack on CentOS 7. Have been fine for about a year.

But today we just saw loading averages hitting through the roof. We checked top didn't see any particular process with high load but there were a stream of very many apache processes running.

Tried to restart apache several times but didn't help.

Finally decided to restart MySQL. This briefly brought down the load but not for long.

Then decided to restart the server. This also just briefly brought down the load.

The logs don't show much of external connections:

$ sudo tail -n 500 /var/log/httpd/access_log | cut -d' ' -f1 | sort | uniq -c | sort -gr
395 ::1
101 127.0.0.1
3 66.102.6.91
1 66.102.6.89

What could be going on? How can we fix this?

Edit 1: More details

Load averages are ranging between 50 and 300. At some point we had this:

 $ uptime
     10:33:28 up 317 days, 22:36,  2 users,  load average: 81.21, 75.67, 75.62

Checking later:

$ uptime
12:41:50 up 50 min,  2 users,  load average: 227.67, 147.99, 75.37

Following Apache spawning hundreds of processes and pushing up load averages we checked config but didn't see ServerLimit or any of the other parameters:

$ less /etc/httpd/conf/httpd.conf | grep ServerLimit
$
Improve this question
3
  • 1
    I would read this lengthy but concise explanation of how load averages work in UNIX /Linux systems. A high load is indicative that there is likely a performance issue somewhere but it's by no means a 'problem to solve'. You're gonna have to figure out whether you have a problem before you can try to fix it. You need to learn some basic diagnostic methods that really arent appropriate for the Q and A format we have here. You should be putting far more effort into solving your own problem before it would be appropriately on-topic for serverfault.
    – user143703
    Aug 27, 2016 at 10:30
  • Also, please be exact as possible. Whatever your load averages that were 'hitting the roof' may be far lower than what I see on a regular basis. Without knowing how many cores your system has, whether it's a vps, or a dedicated server along with what the ACTUAL load average is, we can't even confirm that your load is higher than expected.
    – user143703
    Aug 27, 2016 at 10:34
  • 1
    You need to find out what's going on. There are so many things you can do to debug apache or mysql problems, io performance, memory issues, cpu performance, etc... There are configuration options all over the place that could help/hurt you. Didn't you ever deal with this stuff up until now? It's not something that can be answered in a simple question on SF.
    – Ryan Babchishin
    Aug 27, 2016 at 13:58

1 Answer 1

Reset to default
2

Your system is clearly overloaded. You should install monitoring, use it to gather data and then apply Scientific Method - it's the only way to be sure.

Further reading:

Excessive number of sleeping processes in CentOS - howto diagnose?

Improve this answer
2
  • 1
    I hate you for having a valid answer to a question with none.
    – user143703
    Aug 27, 2016 at 10:48
  • I'm getting bored with writing variations of that answer though :( So far I'm up to 22. Eventually I'll get bored and write a canonical type Q&A and then we can close all the crap against that.
    – user9517
    Aug 27, 2016 at 10:51

Not the answer you're looking for? Browse other questions tagged .