Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a site with around 80K of visitors (daily).

This site is hosted on linode.com and has a nodebalancer (same as a loadbalancer) with 3nodes behind it (2x linode 512 & 1x linode 1024).

Sometimes the traffic is killing me and my servers become unresponsive (/ping doesn't work anymore and forces my nodebalancer to take the node out of rotation).

I'm seeking for a way to find out how to manage the load. I'm already looking in the www.slow.log, to find out possible causes of long lasting processes. Maybe you guys can help me with optimizing my configs for my machines?

My current config:

[www]
user = www-data
group = www-data

listen = 127.0.0.1:9000


pm = dynamic
pm.max_children = 70
pm.start_servers = 10
pm.min_spare_servers = 6
pm.max_spare_servers = 15
pm.process_idle_timeout = 10s;
pm.max_requests = 200
pm.status_path = /status
ping.path = /ping
ping.response = pong
slowlog = /var/log/$pool.log.slow
request_slowlog_timeout = 60
chdir = /
php_admin_value[error_log] = /var/log/fpm-php.www.log
php_admin_flag[log_errors] = on

EDIT

Here is a top of a machine that is building op load

> top - 12:33:41 up 35 days, 44 min, 12 users,  load average: 8.57,
> 12.83, 12.21 Tasks: 193 total,   6 running, 187 sleeping,   0 stopped,   0 zombie Cpu(s): 36.6%us,  4.2%sy,  0.0%ni, 30.1%id,  0.0%wa,  0.0%hi,
> 0.6%si, 28.5%st Mem:   1027516k total,   603048k used,   424468k free,   149040k buffers Swap:   524284k total,   110392k used,   413892k free,
> 54064k cached
> 
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND  
> 31585 www-data  20   0 70600  16m 4388 R   37  1.7   0:09.92 php5-fpm 
> 31558 www-data  20   0 71352  17m 4344 R   33  1.7   0:18.18 php5-fpm 
> 31581 www-data  20   0 66480  13m 4312 S   22  1.3   0:12.08 php5-fpm 
> 31555 www-data  20   0 66220  12m 4344 S   19  1.3   0:16.51 php5-fpm 
> 31578 www-data  20   0 67244  13m 4352 S   19  1.4   0:10.53 php5-fpm 
> 31604 www-data  20   0 66992  13m 4180 S   15  1.3   0:05.31 php5-fpm 
> 31556 www-data  20   0 67244  13m 4324 S   15  1.4   0:16.53 php5-fpm 
> 31557 www-data  20   0 66476  12m 4340 R   13  1.3   0:17.35 php5-fpm 
> 31602 www-data  20   0 66468  13m 4320 S   12  1.3   0:06.59 php5-fpm 
> 31582 www-data  20   0 67244  13m 4328 S   11  1.4   0:11.18 php5-fpm 
> 31579 www-data  20   0 69328  15m 4344 R   11  1.6   0:10.76 php5-fpm 
> 31586 www-data  20   0 66736  13m 4308 S    9  1.3   0:08.17 php5-fpm 
> 31603 www-data  20   0 67504  14m 4324 S    8  1.4   0:06.35 php5-fpm 
> 31580 www-data  20   0 67244  13m 4336 S    7  1.4   0:11.06 php5-fpm 
> 31583 www-data  20   0 66736  13m 4300 S    7  1.3   0:10.80 php5-fpm 
> 31584 www-data  20   0 68024  14m 4356 S    6  1.4   0:10.17 php5-fpm 
> 31587 www-data  20   0 66736  13m 4328 S    6  1.3   0:08.18 php5-fpm 
> 31574 www-data  20   0 57856  52m  816 S    2  5.2   0:01.75 nginx    
> 31634 ward      20   0  2520 1040  736 R    1  0.1   0:00.09 top      
> 15554 ward      20   0  2520  552  264 R    1  0.1  59:04.40 top      
> 1023 root       0 -20     0    0    0 S    0  0.0  18:19.96
> kworker/0:1H                                                          
> 31575 www-data  20   0 57856  52m  808 S    0  5.2   0:00.53 nginx    
> 31607 ward      20   0  8868 1292  772 S    0  0.1   0:00.01 sshd     
> 
>     1 root      20   0  2088  120  120 S    0  0.0   0:46.60 init                                                                                                                                                                                                                                                            
>     2 root      20   0     0    0    0 S    0  0.0   0:03.17 kthreadd

EDIT2: Feedback on solution by growse

I switched the 3 servers to static pools. Looks like the load is better under control now. I also increase the max_requests to 5000.

You can see the pool statics of each server below:

Server1: 1024 machine

pool:                 www
process manager:      static
start time:           30/Dec/2012:12:48:23 +0100
start since:          489
accepted conn:        4311
listen queue:         0
max listen queue:     0
listen queue len:     128
idle processes:       35
active processes:     15
total processes:      50
max active processes: 35
max children reached: 0

Server2: 512 machine

pool:                 www
process manager:      static
start time:           30/Dec/2012:12:45:40 +0100
start since:          709
accepted conn:        11010
listen queue:         0
max listen queue:     9
listen queue len:     128
idle processes:       33
active processes:     7
total processes:      40
max active processes: 40
max children reached: 0

Server3: 512 machine

pool:                 www
process manager:      static
start time:           30/Dec/2012:12:38:25 +0100
start since:          1159
accepted conn:        21645
listen queue:         0
max listen queue:     129
listen queue len:     128
idle processes:       35
active processes:     5
total processes:      40
max active processes: 40
max children reached: 0

All servers are running more stable now with a load around 1-2.

share|improve this question
    
what does the console say on a hanged machine? What do the syslogs say? –  growse Dec 30 '12 at 11:04
    
I just see a high load and a lot of PHP5-fpm processes (top). For now everything is stable, but in a few hours I probably have the same problem again. What should I look for in the syslog? Can't find anything special... –  Ward Loockx Dec 30 '12 at 11:11
    
If you can get top running on the console, and then view it when the system has 'hung', it may give some indication as to what's going on, especially if syslog doesn't say anything useful. Have you adjusted any kernel parameters like vm.swappiness? Also, while I think of it, 70 max_children seems a lot. Your server has 1, maybe two CPUs - maybe lower your child process count a little. (This may be one of those times that if the load is too much for the hardware, then you can manage this by adding more hardware). –  growse Dec 30 '12 at 11:17
    
I didn't adjusted the vm.swapiness parameter. What happens when the max_children are all busy, with new connections? Do they get a fail message (4xx message) or are they queued? I'll try too capture a TOP when this happens.. 80K of daily visitors is too much for these 3 machines? –  Ward Loockx Dec 30 '12 at 11:31
    
I misread 'per day' for 'per hour'. 80K per day is about 1 per second, but obviously you want to be able to handle burst traffic. Incoming connections are queued until there's a child to handle them. As for swappiness, I prefer the OOM-killer to just kill stuff under memory pressure rather than live in a world of kswapd using 99% CPU. But that's me. –  growse Dec 30 '12 at 11:34

1 Answer 1

up vote 2 down vote accepted

I'm going to go out on a limb after thinking about the comments and suggest three things:

  1. Switch from a dynamic pool to a static pool: Assuming you have a single FPM pool and your server isn't doing much else, then there's not much reason to keep scaling up and down the number of processes you're using. Pick a number of children, spin up that many processes and then leave it.

  2. Have far fewer child processes: Ultimately, your VPS has a couple of CPUs at most. Each FPM child process can handle one request at once, others just get queued up behind. All having many processes does is accept many requests from the queue, and then forces them to compete for CPU usage. You might have better results if you have a smaller pool and let burst traffic queue up.

  3. Increase the lifetime of the FPM child processes: you're currently killing each child once it has served 200 requests. This isn't a great many, and a burst of traffic could see a flurry of child processes being killed and respawned, which just adds overhead. Increase this to a larger value (say, 5000) and see what happens.

It's worth pointing out that it's difficult to know if these suggestions will work, because I don't really know what the problem actually is (it could be faulty hardware, or something else entirely). These seem sensible to me: try it and see if it works for you in your environment.

share|improve this answer
    
Are there considerations on hardware vs number of processes I should be aware of? –  Ward Loockx Dec 30 '12 at 11:35
    
I updated my original question with feedback on your answer. Should I increase the number of processes based on the stats? –  Ward Loockx Dec 30 '12 at 12:00
    
From the looks of things, you're yet to increment your max children reached counter on any of the servers, so you're never in a situation where you're queueing a request because you've run out of children. On the flip side, you've got quite a few processes active, which (I think) means they're handling a request. Many processes simultaneously doing work means CPU contention, which may, or may not cause you issues. Biggest issue in these sorts of changes is changing too much, too quickly. I'd leave it a week (at least) on these settings and then see if you're happy, and go from there. –  growse Dec 30 '12 at 19:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.