Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

running out of ideas to explore. First off - let me warn you - I'm a programmer, not a systech :)

Here is the situation.

Dedicated server (LAMP) running a fair amount of sites. mySQL server is on a seperate box.

Last couple weeks, performance has been steadily degrading to the point where I can no longer even remote into the box.

Looking into mod_status, there are a fair amount of processes taking up CPU resources. However, the URLs are all different... there is not a common pattern - so I can't narrow anything down to a particular script that might be getting stuck.

PHP is ran as cgi.

Majority of the sites that are taking a while to run are using the cakephp framework

Restart the server, we are down within a few minutes again...

Crossed an error that said /var/tmp/ was full and couldn't write sessions. However, there was still room? Lack of inodes perhaps? Currently in the process of having someone walk down to the box and clear tmp.

Could the lack of ability to write sessions be causing the php processes to hang forever, and eventually clog everything up?

Any other ideas that I might want to explore? I have been monitoring the sql server to see if it is returning huge datasets in any of the queries, and there is nothing notable in there....

It's only 11:21am here and I already need a drink :)

share|improve this question

migrated from stackoverflow.com Aug 16 '12 at 13:52

This question came from our site for professional and enthusiast programmers.

1  
as you are getting progressively poorer performance this sounds like a memory leak. However I'm no sysadmin so perhaps post this over on serverfault. –  ToonMariner Aug 15 '12 at 18:28
    
processes have urls? –  Marc B Aug 15 '12 at 18:33
    
Ian - thanks, I will post over there as well. Marc - yes, but nothing common.. just random pages on a variety of sites. Only common denominator is that they are sites that are using cakephp. –  Ryan Aug 15 '12 at 18:36

2 Answers 2

I assume its a memory problem.

  1. Apache is eating a lot of RAM.

  2. PHP also has a lot of memory leaks. You should configure it to restart its worker threads after handling some low amount of request (100 is a good number). Look in /etc/init.d/php-cgi (or similar) for a line "PHP_FCGI_MAX_REQUESTS=20" ... that the limit. Also set a reasonable limit for the number of children like "PHP_FCGI_CHILDREN=15". I would also suggest you to use php-fpm if possible, thats much more stable and has less memory leaks.

TODO:

  1. Try to look for killed processes in your syslog (/var/log/syslog or /var/log/messages depending on distribution). There might be such a hint.
  2. To track the problem down, try to use "atop" (process monitor like top, but some more features) and press "p", that accumulates all statistics by process names. Have a look at what's eating up the RSIZE.
share|improve this answer
    
Will look into. Would a lack of memory result in high cpu usage? –  Ryan Aug 15 '12 at 18:37
    
If the server starts swapping, most definitely, yes. –  trion Aug 15 '12 at 18:46
    
Yes, if your memory is full the system starts to use swap space. That means delays in executions of programms. They get blocked until memory is transfered to disk and back... that way the CPU-load goes up too. Have an eye on "swap usage" (at best there shouldnt be any) and WAITIO (in "top" displayed as "1.0%wa" or similar); high WAITIO means that processes wait for IO to finish (may be network/disk but is most times the hard drive). Also a defective hard drive may be the problem. Try "hdparm -t /dev/sda" where /dev/sda is you hdd, if below 10MB/s its probably defective and causes probs. –  Stefan K. Aug 15 '12 at 18:46
    
Got the /var/tmp cleared up... performance seems to have been improving.. Memory doesn't seem to be an issue at the moment. We have 16 gigs on the box. (5 in usage at the moment). Continuing to investigate. –  Ryan Aug 15 '12 at 19:41
    
Ok, what is "top" saying about WAITIO? What is the result of "hdparm"? –  Stefan K. Aug 15 '12 at 20:08

You really need to look from inside the box instead of outside, so see what resource is being consumed.

My guess would be that apache's process pool is exhausted (so no one can connect) or physical memory is exhausted (so performance falls off a cliff).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.