2

running out of ideas to explore. First off - let me warn you - I'm a programmer, not a systech :)

Here is the situation.

Dedicated server (LAMP) running a fair amount of sites. mySQL server is on a seperate box.

Last couple weeks, performance has been steadily degrading to the point where I can no longer even remote into the box.

Looking into mod_status, there are a fair amount of processes taking up CPU resources. However, the URLs are all different... there is not a common pattern - so I can't narrow anything down to a particular script that might be getting stuck.

PHP is ran as cgi.

Majority of the sites that are taking a while to run are using the cakephp framework

Restart the server, we are down within a few minutes again...

Crossed an error that said /var/tmp/ was full and couldn't write sessions. However, there was still room? Lack of inodes perhaps? Currently in the process of having someone walk down to the box and clear tmp.

Could the lack of ability to write sessions be causing the php processes to hang forever, and eventually clog everything up?

Any other ideas that I might want to explore? I have been monitoring the sql server to see if it is returning huge datasets in any of the queries, and there is nothing notable in there....

It's only 11:21am here and I already need a drink :)

3
  • 1
    as you are getting progressively poorer performance this sounds like a memory leak. However I'm no sysadmin so perhaps post this over on serverfault. Commented Aug 15, 2012 at 18:28
  • processes have urls?
    – Marc B
    Commented Aug 15, 2012 at 18:33
  • Ian - thanks, I will post over there as well. Marc - yes, but nothing common.. just random pages on a variety of sites. Only common denominator is that they are sites that are using cakephp.
    – Ryan
    Commented Aug 15, 2012 at 18:36

2 Answers 2

4

I assume its a memory problem.

  1. Apache is eating a lot of RAM.

  2. PHP also has a lot of memory leaks. You should configure it to restart its worker threads after handling some low amount of request (100 is a good number). Look in /etc/init.d/php-cgi (or similar) for a line "PHP_FCGI_MAX_REQUESTS=20" ... that the limit. Also set a reasonable limit for the number of children like "PHP_FCGI_CHILDREN=15". I would also suggest you to use php-fpm if possible, thats much more stable and has less memory leaks.

TODO:

  1. Try to look for killed processes in your syslog (/var/log/syslog or /var/log/messages depending on distribution). There might be such a hint.
  2. To track the problem down, try to use "atop" (process monitor like top, but some more features) and press "p", that accumulates all statistics by process names. Have a look at what's eating up the RSIZE.
17
  • Will look into. Would a lack of memory result in high cpu usage?
    – Ryan
    Commented Aug 15, 2012 at 18:37
  • If the server starts swapping, most definitely, yes.
    – trion
    Commented Aug 15, 2012 at 18:46
  • Yes, if your memory is full the system starts to use swap space. That means delays in executions of programms. They get blocked until memory is transfered to disk and back... that way the CPU-load goes up too. Have an eye on "swap usage" (at best there shouldnt be any) and WAITIO (in "top" displayed as "1.0%wa" or similar); high WAITIO means that processes wait for IO to finish (may be network/disk but is most times the hard drive). Also a defective hard drive may be the problem. Try "hdparm -t /dev/sda" where /dev/sda is you hdd, if below 10MB/s its probably defective and causes probs.
    – Stefan K.
    Commented Aug 15, 2012 at 18:46
  • Got the /var/tmp cleared up... performance seems to have been improving.. Memory doesn't seem to be an issue at the moment. We have 16 gigs on the box. (5 in usage at the moment). Continuing to investigate.
    – Ryan
    Commented Aug 15, 2012 at 19:41
  • Ok, what is "top" saying about WAITIO? What is the result of "hdparm"?
    – Stefan K.
    Commented Aug 15, 2012 at 20:08
0

You really need to look from inside the box instead of outside, so see what resource is being consumed.

My guess would be that apache's process pool is exhausted (so no one can connect) or physical memory is exhausted (so performance falls off a cliff).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .