1

I get the follow error in the nginx log

[error] 17734#0: *6643 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: [cut], server: [cut], request: "GET /venues HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "[cut]"

I have a dedicated box with 8 gb ram, quad core chip. Good server. Nginx, php-fpm & mysql all latest versions running under ubuntu 10.04

I only get this when I stress test the server with siege. If I increase the number of concurrent connections to 100, I can get up to 20% of all requests to fail.

Furthermore, I don't get this on pages that have no mysql queries. And only a few failures on pages with moderate number of queries. Bit, I'm not sure if that's got to do anything with it.

I have a feeling this is something to do with php. But I can't figure it out.

Any suggestions of where to even start looking?

Update: and the php error log is silent. No record of anything going wrong

2
  • I've seen many of these threads pop up in the php-highload-en group. One guys problem was hitting the mysql connection limit durring siege . Most of the time they recommend profiling the application and its sql queries. You could try posting there: groups.google.com/group/highload-php-en
    – iainlbc
    Aug 18, 2011 at 23:24
  • What about the nginx error log? And the MySQL normal log as well as the slow query log? Is query caching in play at all? What is the nginx.conf like? What is the php config like?
    – JayMcTee
    Nov 11, 2015 at 9:23

2 Answers 2

1

Most probably you ran out of php-fpm workers. The log was silent because nothing was wrong in the code itself - workers just were busy processing your requests. If you did not get this on pages without MySQL queries, the bottleneck was the MySQL DB. You should identify long-running queries (using mytop or slow log feature or maybe some custom PHP logging around your SQL processing) and optimize them. Of course, "long queries" are actually quite short in high-load we world. Even 200ms query is long enough to put your server on its knees.

3
  • Does php-fpm seriously just silently drop connections when it's pool is fully occupied? That's horrendous.
    – womble
    Aug 18, 2011 at 23:08
  • @womble When load comes in peaks it depends on a TCP backlog size I guess, but when load is constantly high the backlog will end sooner or later, so this behavior is quite predictable.
    – Alex
    Aug 18, 2011 at 23:14
  • 1
    Sure, sometimes you can't handle something, and dropping a connection is possibly the best option. It's the silence I can't fathom. If you're going to do something out of the ordinary, you have to log it. No exceptions.
    – womble
    Aug 18, 2011 at 23:42
-1

this may be resolved. i reported a similar issue with opening and reusing persistent tcp sockets under almost no load at all. there's now a patch landed for this in git:

https://groups.google.com/forum/#!topic/highload-php-en/qGu3Eaifj9s

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .