Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've run into a bit of a brick wall with troubleshooting my dedicated websever. Recently, my website spiked with the amount of request/per second and it crashed.

The box original had 8GB of RAM, 8 Core Xeon E3-1230, 1TB 7,200 RPM Disk (No Raid), 100Mbit dedicated networking.

After the spike, I increased the RAM to 24GB in order to support more concurrent users.

Apache seems to handle well, even with 3000 concurrent users, it will return HTML and static content very quickly (un-cached).

To further test the differences between Apache/HTML and Apache/PHP, I ran ab.

Both test.html and test.php have exactly the same static content, the PHP does not call any includes and does not connect with MySQL.

HTML Test

ab -n 500 -c 50 http://www.~~.com/test.html

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      252  375 190.3    276    1399
Processing:   254  354 121.5    282     657
Waiting:      253  353 121.4    280     653
Total:        510  730 231.7    573    1675

PHP Test

ab -n 500 -c 50 http://www.~~~.com/test.php

Connect:      248  275  51.1    267    1316
Processing:   256 4167 6210.2   2262   41489
Waiting:      253 4166 6210.2   2262   41489
Total:        509 4442 6212.4   2523   41754

Pingdom also reports a long wait time when accessing a PHP script. enter image description here

I'm getting a similar result on WebPageTest.org, although better, the first time to byte is F:

           Load Time   **First Byte**   Start Render    DOM Elements    Time    Requests    Bytes In    Time    Requests    Bytes In
First View  2.061s  **0.839s**  0.000s  55  2.061s  20  428 KB  2.061s  20  430 KB

Here is my top results: enter image description here

I/O Test

Under heavy load, the wa% can increase to 95% for a few milliseconds.

I ran iostat during load:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           8.37    0.00    5.18    0.56    0.00   85.88

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00    45.50  3.00 48.00   136.00   748.00    17.33     3.05   59.76   2.53  12.90
sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb3              0.00    45.50  3.00 48.00   136.00   748.00    17.33     3.05   59.76   2.53  12.90

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.00    0.00    3.56    0.69    0.00   91.75

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               6.00   118.50  9.50 21.00   996.00  1116.00    69.25     0.29    9.44   1.66   5.05
sdb1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb2              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb3              6.00   118.50  9.50 21.00   996.00  1116.00    69.25     0.29    9.44   1.66   5.05

To me, that doesn't look bad, but I may be missing something.

I'm using FCGI

<IfModule mod_fcgid.c>
FcgidMaxRequestLen 1547483648
FcgidMaxRequestInMem 52485760
FcgidIdleScanInterval 15000
FcgidBusyTimeout 15000
FcgidProcessLifeTime 7200
FcgidConnectTimeout 1800
FcgidIOTimeout 1800
PHP_Fix_Pathinfo_Enable 1
FcgidMaxRequestsPerProcess 1000
</IfModule>

And here is my Apache Conf (I'm using 2.4.x)

Timeout 60
TraceEnable Off
ServerSignature Off
ServerTokens ProductOnly
FileETag None
StartServers 10
<IfModule prefork.c>
MinSpareServers 5
MaxSpareServers 15
</IfModule>
<IfModule itk.c>
MinSpareServers 5
MaxSpareServers 15
</IfModule>
ServerLimit 2200
MaxRequestWorkers 2000
MaxConnectionsPerChild 15000
KeepAlive On
KeepAliveTimeout 1
MaxKeepAliveRequests 2000

I've looked into my Apache Error Logs and Access Logs. Nothing strange to report.

I'm really scratching my head here.

I've tried turning off the firewall.

I've tried increasing Max Connections.

I've optimized mySQL and removing many slow queries (that were >0.5s).

What else can I do, is there anything I can use to help identify problems? Any help would be greatly appreciated.

P.S:

It's worth noting, that even when the server is being heavily accessed, PHPMyAdmin and cPanel are still very responsive. Nothing else seems to lag except the PHP on the website.

share|improve this question
    
add httpd status out put (if you use cpanel ) and what about internet conecction, you have a 10 mbits port 100mbits port or 1gbit, and what about usage check it whit iptraf, and you have 4.5 % of IO WAIT on screan shot, maybe a slow disk is a problem, you dont have raid, online one 7200 rpm disk, that is slow for a server. –  Skamasle Jan 4 at 5:02
    
Yea I was also thinking that also, but I guess if I can minimise any hardware upgrade expense, it would help immensely. I have a dedicated 100mbit port. –  Moe Jan 4 at 7:28
    
Is phpmyadmin running on the same server but in a different VirtualHost directive? If so, then likely there is something in the website's VirtualHost (or .htaccess?) causing the problem (eg allow/deny using a hostname and you have a slow ip lookup). Could also be a user.ini file for your website directory causing weird php startup issues. –  DerfK Jan 4 at 16:13

2 Answers 2

up vote 1 down vote accepted

I think your problem is simply your hard disk with it's access times. Apache can cache html-pages in memory, but php-scripts are not cached; the need to be executed every time again. Therefore the php interpreter is called and it reads the script from the hdd. This takes a lot of time. The most slowest thing on your server is your HDD. On my pc, there is an extreme difference between applications starting from my SSD and my HDD (the SSD is up to 10 times faster!). If your HDD works, then the latency for executing an php script may increase drastically.

Possible solutions: get an SSD (maybe a small one just for often accessed data such as scripts) and minimize script calls (and HDD accesses) on your server. Make sure, the filesystem is defragmented (usually done automatically). If your php-script creates often the same contents, try caching them into an html file.

This answer could also help you: http://stackoverflow.com/questions/4181865/apache-php-caching

share|improve this answer
1  
Thanks for this. I installed APC with little to no improvement. Upon the recommendation of the next answer, I installed varnish cache. I was skeptical at first, and setting it up wasn't as straight forward as the tutorials made it out to be. But once I got it working. Wow. My server is now a rocket and my server load rarely passes 1! Madness! –  Moe Jan 5 at 11:22

HTML file you're testing is just a plain file, all that Apache has to do is do a few system calls (open, read) and then serve its content.

PHP OTOH is actually quite a "heavy" option: it's an entire interpreter (bytecode compiler?). And since you're using concurrent testing (-c) who knows how well the requests are multiplexed to it? This does not have to be apache problem at all, but rather PHP's problem.

What I'd do:

  1. Switch to Apache MPM (multi-threaded in multiple processes).

  2. Do sequential test (no multiple concurrent requests), compare.

  3. Run valgrind or some such on apache process and see where most CPU time is spent (apache or PHP).

  4. run the same test but serving this page via nginx. Since nginx is based on async model and very fast, then if you get similar results, PHP is the culprit.

  5. Finally, you could install Zend Optimizer (30-day trial is free) or smth like this: https://github.com/zendtech/ZendOptimizerPlus.

Really, comparing serving a static file to dynamically generated webpage is sort of apples to oranges comparison. None of the typical solutions (Python mod-apache, Django, PHP, etc) are going to be very fast in this regard, at least when compared to serving a static file. Node.js is an exception perhaps, due to sort of "low level" programming webpage directly in async model.

P.S. you have not quoted php.ini content. Post it and/or tweak it.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.