Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I get the follow error in the nginx log

[error] 17734#0: *6643 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: [cut], server: [cut], request: "GET /venues HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "[cut]"

I have a dedicated box with 8 gb ram, quad core chip. Good server. Nginx, php-fpm & mysql all latest versions running under ubuntu 10.04

I only get this when I stress test the server with siege. If I increase the number of concurrent connections to 100, I can get up to 20% of all requests to fail.

Furthermore, I don't get this on pages that have no mysql queries. And only a few failures on pages with moderate number of queries. Bit, I'm not sure if that's got to do anything with it.

I have a feeling this is something to do with php. But I can't figure it out.

Any suggestions of where to even start looking?

Update: and the php error log is silent. No record of anything going wrong

share|improve this question
    
I've seen many of these threads pop up in the php-highload-en group. One guys problem was hitting the mysql connection limit durring siege . Most of the time they recommend profiling the application and its sql queries. You could try posting there: groups.google.com/group/highload-php-en –  iainlbc Aug 18 '11 at 23:24
add comment

2 Answers

Most probably you ran out of php-fpm workers. The log was silent because nothing was wrong in the code itself - workers just were busy processing your requests. If you did not get this on pages without MySQL queries, the bottleneck was the MySQL DB. You should identify long-running queries (using mytop or slow log feature or maybe some custom PHP logging around your SQL processing) and optimize them. Of course, "long queries" are actually quite short in high-load we world. Even 200ms query is long enough to put your server on its knees.

share|improve this answer
    
Does php-fpm seriously just silently drop connections when it's pool is fully occupied? That's horrendous. –  womble Aug 18 '11 at 23:08
    
@womble When load comes in peaks it depends on a TCP backlog size I guess, but when load is constantly high the backlog will end sooner or later, so this behavior is quite predictable. –  Alex Aug 18 '11 at 23:14
1  
Sure, sometimes you can't handle something, and dropping a connection is possibly the best option. It's the silence I can't fathom. If you're going to do something out of the ordinary, you have to log it. No exceptions. –  womble Aug 18 '11 at 23:42
add comment

this may be resolved. i reported a similar issue with opening and reusing persistent tcp sockets under almost no load at all. there's now a patch landed for this in git:

https://groups.google.com/forum/#!topic/highload-php-en/qGu3Eaifj9s

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.