Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have nginx as a front end proxy and apache with mod_fcgid.

This box has 24GB RAM and 16 cores. I shouldn't be bottlenecking.

There seems to be a bottleneck. Once it hits 7000 requests it hangs before continuing.

EDIT 3: Can someone explain PHP_FCGI_CHILDREN? I set it to 1 and stuff seems to work better. Does setting it higher better? It really spans a lot of processes like crazy higher it is.

EDIT: error log wise:

(104)Connection reset by peer: mod_fcgid: error reading data from FastCGI server

EDIT - QUESTION 2 Because I am using nginx as a frontend proxy. Is it possible I am maxing out all my ports thus can no longer send to the backend(apache)? I also noticed that PHP procs die fast

16409 USER 20   0     0    0    0 Z 27.2  0.0   0:00.82 [php] <defunct>

Even with something like this it fails:

/usr/local/apache/bin/ab -n 1000000 -c 10 http://IP/index.php
apr_poll: The timeout specified has expired (70007)
Total of 23998 requests completed

My results:

/usr/local/apache/bin/ab -n 10000 -c 100 http:///index.php

Document Path:          /index.php
Document Length:        55764 bytes

Concurrency Level:      100
Time taken for tests:   52.076 seconds
Complete requests:      10000
Failed requests:        17
   (Connect: 0, Receive: 0, Length: 17, Exceptions: 0)
Write errors:           0
Non-2xx responses:      17
Total transferred:      558563436 bytes
HTML transferred:       556703249 bytes
Requests per second:    192.03 [#/sec] (mean)
Time per request:       520.764 [ms] (mean)
Time per request:       5.208 [ms] (mean, across all concurrent requests)
Transfer rate:          10474.46 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.4      0       6
Processing:     3  436 3978.0      3   52069
Waiting:        2  436 3978.0      3   52069
Total:          3  436 3978.1      3   52073

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      4
  75%      4
  80%      4
  90%      5
  95%      5
  98%   1018
  99%  29055
 100%  52073 (longest request)

/etc/sysctl.conf

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_reordering = 5
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 3
net.ipv4.tcp_max_syn_backlog = 2048
net.core.netdev_max_backlog = 1024
net.core.hot_list_length = 256
net.ipv4.tcp_max_tw_buckets = 360000
net.core.rmem_default = 65535
net.core.rmem_max = 8388608
net.ipv4.tcp_rmem = 4096 87380 8388608
net.core.wmem_default = 65535
net.core.wmem_max = 8388608
net.ipv4.tcp_wmem = 4096 65535 8388608
net.ipv4.tcp_mem = 8388608 8388608 8388608
net.core.optmem_max = 40960
net.ipv4.netfilter.ip_conntrack_max = 32768
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_orphan_retries = 1
net.ipv4.tcp_fin_timeout = 25
net.ipv4.tcp_max_orphans = 8192
net.ipv4.ip_local_port_range = 2768    61000
fs.file-max = 209708

/usr/local/cpanel/cgi-sys/php5

export PHP_FCGI_CHILDREN=0
exec /usr/bin/php

/usr/local/apache/conf/php.conf

LoadModule fcgid_module modules/mod_fcgid.so
MaxRequestsPerProcess 1000
FcgidMaxProcesses 200
FcgidProcessLifeTime 3600
MaxProcessCount 200
FcgidIOTimeout 2000
FcgidIdleTimeout 60
FcgidIdleScanInterval 30
FcgidBusyTimeout 300
FcgidBusyScanInterval 80
ErrorScanInterval 3
ZombieScanInterval 3
FcgidMinProcessesPerClass 5
FcgidMaxProcessesPerClass 150
DefaultMinClassProcessCount 5
DefaultMaxClassProcessCount 150
IPCCommTimeout 40
IPCConnectTimeout 10
MaxRequestLen 1073741824
AddHandler fcgid-script .php5 .php4 .php .php3 .php2 .phtml
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .php5
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .php4
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .php
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .php3
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .php2
FCGIWrapper /usr/local/cpanel/cgi-sys/php5 .phtml

Apache

Timeout 5
TraceEnable Off
ServerSignature Off
ServerTokens ProductOnly
FileETag None
StartServers 15
<IfModule prefork.c>
MinSpareServers 5
MaxSpareServers 10
</IfModule>
ServerLimit 256
MaxClients 150
MaxRequestsPerChild 1000
KeepAlive Off
KeepAliveTimeout 1
MaxKeepAliveRequests 1

httpd -V

Server version: Apache/2.2.23 (Unix)
Server built:   Jan 29 2013 11:18:48
Cpanel::Easy::Apache v3.16.7 rev9999
Server's Module Magic Number: 20051115:31
Server loaded:  APR 1.4.6, APR-Util 1.4.1
Compiled using: APR 1.4.6, APR-Util 1.4.1
Architecture:   64-bit
Server MPM:     Event
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/experimental/event"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/usr/local/apache"
 -D SUEXEC_BIN="/usr/local/apache/bin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

nginx error log

2013/02/06 18:12:50 [notice] 15123#0: signal 17 (SIGCHLD) received
2013/02/06 18:12:50 [notice] 15123#0: worker process 15127 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: worker process 15128 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: worker process 15129 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: worker process 15131 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: worker process 15137 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: worker process 15139 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: signal 29 (SIGIO) received
2013/02/06 18:12:50 [notice] 15123#0: signal 17 (SIGCHLD) received
2013/02/06 18:12:50 [notice] 15123#0: worker process 15124 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: signal 29 (SIGIO) received
2013/02/06 18:12:50 [notice] 15123#0: signal 17 (SIGCHLD) received
2013/02/06 18:12:50 [notice] 15123#0: worker process 15136 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: signal 29 (SIGIO) received
2013/02/06 18:12:50 [notice] 15123#0: signal 17 (SIGCHLD) received
2013/02/06 18:12:50 [notice] 15123#0: worker process 15125 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: signal 29 (SIGIO) received
2013/02/06 18:12:50 [notice] 15123#0: signal 17 (SIGCHLD) received
2013/02/06 18:12:50 [notice] 15123#0: worker process 15130 exited with code 0
2013/02/06 18:12:50 [notice] 15123#0: signal 29 (SIGIO) received
share|improve this question
    
Probably not what you are looking for but If you can replace mod_fcgid with php-fpm I would use the latter. Gives me more reliable results. It works with mod_fastcgi tutorial or omit Apache and use php-fpm directly with nginx. –  mt_ Feb 6 '13 at 20:24
    
How hard is it to replace mod_fcgid with php-fpm? –  Tiffany Walker Feb 6 '13 at 20:39
    
Take a look at the tutorial. php-fpm has a lot of config files and options. There is some documentation here: php-fpm.org/wiki/Main_Page (probably outdated) and here php.net/manual/en/install.fpm.php. How hard? I think it depends on your application. If fcgid works for you I would not worry to switch. Plesk and Cpanel Integration does not exist, so you are on your own with php-fpm. The results you get could be caused by the ab benchmarking tool and not mod_fcgid. So don't switch if it's not critical. –  mt_ Feb 6 '13 at 20:45
    
What do you mean by "Once it hits 7000 requests it hangs"? Your MaxClients is 150. Are you saying that if you send 7000 requests in /serial/ fashion to your webserver once it hits the 7000th request it just hangs? e.g. ab -c 1 -n 7000 example.com –  HTTP500 Feb 6 '13 at 21:25
    
It went from 1000-7000 real quick and then it just sat on 7000/8000 before finishing. –  Tiffany Walker Feb 6 '13 at 21:26
show 8 more comments

3 Answers

up vote 0 down vote accepted

If you already use ngix for a reverse proxy and I assume also for serving static files, then mod_fcgi is just a complication — just use apache with mod_php and reasonably high maxclients and serverlimit (something like 1024).

share|improve this answer
    
So I really don't need mod_fcgid for performance in anyway? So running as DSO would be faster and quicker and easier? –  Tiffany Walker Feb 7 '13 at 0:09
    
I'm pretty sure you don't. The problem with running apache+php directly is that keepalive connections and static requests use resources that are very limited, like httpd processes in prefork-mpm. And you have to use prefork if you need PHP. But when you hava a ngix as a reverse proxy and it is serving static files, then you have no problems. I even run apache in worker-mpm mode as a reverse proxy for several other apache instances, each running another PHP site with mod_php and prefork and each running as it's own UID for security isolation. –  Tometzky Feb 7 '13 at 7:46
add comment

Are you running the benchmark on a separate machine to the server? What's the network link speed on the server? You could be maxing out the network bandwidth.

Transfer rate:          10474.46 [Kbytes/sec] received

This looks suspiciously like 10MB/s

share|improve this answer
    
This was being ran on the same machine –  Tiffany Walker Feb 6 '13 at 21:33
add comment

The percentage of requests served within 5 seconds is 95, then all of a sudden jumps to 1018 seconds to server just 98%. It also happens that your Apache Timeout is set to 5 seconds. I think the default Timeout value for Apache is generally set to be greater than 120 seconds by default. Is there a reason why you set it to 5 seconds? Have you considered increasing it?

 95%      5
 98%   1018
share|improve this answer
    
Tried and same results. –  Tiffany Walker Feb 6 '13 at 21:51
    
Your MaxClients seems very low for a server of such capacity. May be you need to increase that, then restart the service - just to remind you - many of us forget that. –  Daniel t. Feb 6 '13 at 22:32
    
Set 1024. It looks to be more PHP related then Apache related. –  Tiffany Walker Feb 6 '13 at 22:39
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.