PHP problem / Server unstable

Question

I'm using the PHP script Openx to run ad campaigns. We started getting more traffic and the server sometimes becomes unstable. When I check top command I see how httpd and PHP is running and then for couple seconds the PHP process stops and then again there.

Problem is in slow responding. Pages show up very slow. Looks like server hits the limit of something in peak time. Please, can you suggest anything?!

Server is running with: fcgi; Linux CentOS 5.5; Apache 2.2; PHP 5.2.9;

I made some modifications to httpd.conf as follows

Timeout 10
TraceEnable On
ServerSignature Off
ServerTokens Full
FileETag All
StartServers 50
<IfModule prefork.c>
  MinSpareServers 50
  MaxSpareServers 100
</IfModule>
ServerLimit 512
MaxClients 512
MaxRequestsPerChild 50
KeepAlive On
KeepAliveTimeout 1
MaxKeepAliveRequests 500

System information

Server load 0.83 (6 CPUs)   
Memory Used 6.08% (997296 of 16410212)  
Swap Used   0.00% (0 of 2096472)    

 1  0      0 4105112 324280 11092200    0    0     0   252 3336 7169 14  4 81  0  0
 1  0      0 4098736 324280 11089104    0    0     0   240 3287 6792 13  4 83  0  0
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 4119496 324280 11082052    0    0     0   186 3221 6946 13  4 83  0  0
 2  0      0 4122120 324280 11086072    0    0     0   242 3212 6904 13  4 83  0  0
 2  0      0 4101952 324280 11085588    0    0     0   174 3442 6774 14  4 82  0  0
 0  0      0 4078672 324280 11090232    0    0     0   224 3465 7558 15  4 80  0  0
 5  0      0 4070644 324280 11100076    0    0     0   240 3603 7921 15  4 81  0  0
 1  0      0 4105108 324280 11079192    0    0     0   535 3507 6977 13  4 83  0  0
 1  0      0 4074172 324280 11095496    0    0     0   244 3280 6592 13  4 82  0  0
 1  0      0 4067656 324280 11105696    0    0     0   190 3331 6654 12  4 85  0  0
 1  0      0 4096528 324280 11093160    0    0     0   188 3068 5648 11  3 86  0  0
 1  0      0 4099368 324280 11092972    0    0     0   174 3130 6325 12  4 84  0  0
 1  0      0 4105312 324284 11091888    0    0     0   225 3397 7064 14  4 82  0  0
 1  0      0 4101616 324284 11096192    0    0     0   234 3517 7039 14  4 82  0  0
 3  0      0 4062576 324284 11103648    0    0     0   372 3547 7263 14  4 82  0  0
 1  0      0 4080312 324284 11097940    0    0     0   243 3180 6623 13  4 83  0  0
 6  0      0 4086780 324284 11102572    0    0     0   174 3254 6390 13  4 83  0  0
 1  0      0 4110312 324284 11089576    0    0     0   181 3198 6403 13  4 83  0  0
 0  0      0 4094080 324284 11099016    0    0     0   220 3141 6949 12  4 84  0  0
 2  0      0 4095848 324284 11096992    0    0     0   226 3043 6483 13  4 83  0  0
 0  0      0 4107396 324288 11086824    0    0     0   264 3250 6551 13  5 82  0  0
 1  0      0 4103260 324288 11094204    0    0     0   364 3188 6066 12  4 84  0  0
 0  0      0 4112904 324288 11095512    0    0     0   179 3155 6355 12  4 84  0  0
 1  0      0 4118736 324288 11098424    0    0     0   241 2929 5381 11  3 86  0  0
 1  0      0 4121464 324292 11089684    0    0     0   213 3076 5939 11  4 85  0  0
 0  0      0 4109180 324292 11103712    0    0     0   274 3076 6172 14  4 82  0  0
 1  0      0 4103344 324292 11107400    0    0     0   182 3186 6301 12  3 84  0  0
 2  0      0 4112200 324292 11100828    0    0     0   184 3234 6435 13  4 83  0  0
 1  0      0 4140368 324292 11087284    0    0     0   345 2944 5851 10  3 86  0  0
 2  0      0 4135768 324292 11082848    0    0     0   219 3390 6779 13  4 83  0  0
 1  0      0 4109100 324292 11099612    0    0     0   237 3086 6635 13  4 83  0  0
 2  0      0 4119308 324292 11089632    0    0     0   197 3203 6504 13  4 83  0  0
 2  0      0 4103316 324292 11089996    0    0     0   212 3322 6576 13  4 83  0  0
 2  0      0 4085044 324292 11103768    0    0     0   239 3313 6924 13  4 83  0  0
 2  0      0 4099672 324292 11100764    0    0     0   226 3063 6233 12  4 84  0  0
 0  0      0 4126776 324292 11088080    0    0     0   289 3361 6555 13  4 84  0  0
 2  0      0 4104824 324292 11090908    0    0     0   231 3228 7022 14  4 82  0  0
 0  0      0 4099456 324296 11096684    0    0     0   224 2936 5995 12  3 85  0  0
 0  0      0 4117268 324296 11087676    0    0     0   228 3220 6257 11  4 85  0  0
 2  0      0 4104404 324296 11098316    0    0     0   234 3347 6693 12  4 84  0  0
 2  0      0 4088440 324296 11106628    0    0     0   228 3379 7079 14  4 82  0  0
 2  0      0 4104672 324300 11103368    0    0     0   185 3012 5821 12  3 85  0  0
 0  0      0 4108812 324300 11104884    0    0     0   268 3172 6914 14  4 82  0  0

Answer 1

By "fcgi" do you mean mod_fastcgi or mod_fcgid? If you're using mod_fcgid, are you using the wrapper script from apache's documentation, or at least having apache set the required environment variables? If there's a configuration mismatch between mod_fcgid and PHP, then the php executable will exit when apache isn't ready for it to exit, which seems to be what you are saying when you say

When I check top command i see how httpd and php is running and then for couple seconds php process stops and then again there.

One other thing with mod_fcgid that needs to be properly configured in PHP is that PHP's children must be disabled since mod_fcgid expects to handle all of the process spawning by itself. See the PHP information in the documentation for an example wrapper script and configuration.

Answer 2

I'm afraid you're not giving us a lot of information here. You need to find where the bottleneck is, and for that you'll need data. Do you have a proper monitoring system collecting statistics on how your processes are behaving and how much resources you are using?

How much RAM and/or disk swap are you using? Can you paste the output of vmstat 5 when both the system is behaving well and when it's slowing down?

One very simple and easy to install monitoring system is "sar". In a Debian or Ubuntu system it can be installed with apt-get install sysstat. Then after a few minutes you can run sar -A to see the data it collected.

All that said, Google shows lots of websites with information on tuning the performance of openx. They might help.

Answer 3

Sorry, no answer, but let me give you an advice. OpenX is a problem itself. You should avoid it.

It's so bloated I can't believe it. I've just downloaded it again to re-check. 45 goddamn Mb just for an admin interface and a script that serves ads? No way.

Relatively recently, I've had an unpleasant experience of using it for some time on a real site with a moderate traffic. It's slow. The maintenance script issues errors and warnings though set up according to the manual, and the whole thing works only by some magic. For example, I had to run the maintenance several times in sequence if I wanted to forcefully rebuild the ads cache (in the admin interface, it's said that the ads refresh right after you save the settings, if I remember correctly, and it doesn't work). Half of advanced settings in the admin interface don't work either.

As a developer, I must notice that they use MDB2 lib to query the database. I've seen the tests - it's one of the slowest PHP DB libs there are out there. And they rolled their own class hierarchy on top of it. I sure hope they use this fatty sandwich only in the admin interface, not in serving the ads.

There was time when it was pretty decent, that was when it was called something like MaxAds. No more.

Answer 4

But if you really want to know what's happening, you might want to look at MySQL queries for some time. Like, issue a 'show processlist' or 'show full processlist' query to the MySQL server and repeat it until you see a query that runs for more time than is reasonable. If you got any luck with your own applications, the query probably will be one of OpenX's.

Another alternative is to look at MySQL slow query log. You'll have to figure out the location of the log file yourself since I'm not familiar with its usual setup. It might be mentioned in the MySQL manual.

Answer 5

• Disable persistent MySQL queries (php.ini)
• Add caching software such as Eaccelerator or Zend to increase PHP performance
• Consider using LiteSpeed or Nginx etc instead of Apache to increased performance
• Monitor disk performance using 'iostat' to try and identify disk performance bottlenecks
• Optomise mySQL by editing the configuration in /etc/my.cnf
• Try the following settings for httpd.conf as a base to get started

KeepAlive Off MaxKeepAliveRequests 100 KeepAliveTimeout 5 MinSpareServers 15 MaxSpareServers 25 StartServers 10 MaxClients 150 MaxRequestsPerChild 0

If you're still running into issues, post a full output of 'top' 'free' and 'iostat'