3

I wrote a RESTful web service to be the backbone behind a mobile app using PHP and the Restler library. It's running on my dev server running Windows Server 2008 R2, PHP 5.3.5, Apache 2.2.17, and MySQL 5.5.8. Just for giggles I decided to benchmark my dev server and ran into a possible configuration issue

If I run ab -k -n 1000 -c 50 http://myproductionserver.com/something/restful through the windows CLI I usually get stopped between 300-700 requests. The error says "apr_socket_recv: An existing connection was forcibly closed by the remote host. (730054). Total of X requests completed."

I figured I would check my Apache Error logs. Not Helpful. Here's the error I invoked with ab:

[Fri Feb 24 17:23:27 2012] [notice] Parent: child process exited with status 255 -- Restarting.
[Fri Feb 24 17:23:27 2012] [notice] Apache/2.2.17 (Win32) PHP/5.3.5 configured -- resuming normal operations
[Fri Feb 24 17:23:27 2012] [notice] Server built: Oct 18 2010 01:58:12
[Fri Feb 24 17:23:27 2012] [notice] Parent: Created child process 768
[Fri Feb 24 17:23:28 2012] [notice] Disabled use of AcceptEx() WinSock2 API
[Fri Feb 24 17:23:28 2012] [notice] Child 768: Child process is running
[Fri Feb 24 17:23:28 2012] [notice] Child 768: Acquired the start mutex.
[Fri Feb 24 17:23:28 2012] [notice] Child 768: Starting 64 worker threads.
[Fri Feb 24 17:23:28 2012] [notice] Child 768: Listening on port 80.

So when I got home, I ran the same test using my local ip- thinking it might be a network issue. Same thing happened. I've been searching for quite a while with not a lot of answers. I found a very similar problem and the resolution was to use "Win32DisableAcceptEx" in the httpd.conf file saying the issue is actually a bug with PHP .dlls. I disabled all of them but the basic MySQLi. As you can tell from the logs above, neither did much of anything.

Anyone have any ideas? Am I seriously crashing the server with 1000 requests? Let me know if you need more info! I know it's just a dev server sitting at home- but it mimics my production server. I'll need to have this sorted out before moving forward. I appreciate the assistance!

Improve this question
6
  • I think you also want to set at least "ab -v 3" to see what the response codes are from apache. 408 indicates a request timeout. But as you have also set -k for keepalives ab is expecting to reuse those 50 connections, but apache will force close the keepalive if it is idle for 5 seconds. So there might be some odd interaction causing that. try it without -k setting and see what happens, also try the -r flag to ab with -v set to 3 or 4, as without -r ab is exiting on the first error
    – Tom
    Feb 25, 2012 at 2:45
  • All 200s then "Send request failed!" and the same error message. that was with -v 3 and -k flags set. Without -k I get 100 requests before failure, and all 200s leading up to it. With -r set, i got 880 / 1000 requests completed, and it would spam the same error message periodically. With -r set and -v not set I get "apr_poll: an operation was attempted on something that is not a socket".-v 4 and -r is the same error.
    – Kevin
    Feb 25, 2012 at 3:10
  • I would start from known good setting for ab, and increase the concurrent connections and total until you start to see errors.
    – Tom
    Feb 25, 2012 at 3:24
  • try ab -c5 -n100 http://" and work up, try any find what the level is for concurrent requests. There is a msdn article here (blogs.msdn.com/b/malarch/archive/2006/06/26/647993.aspx) which gives a scenario in which the SynProtection mechanism which is used to protect against "syn DOS" attacks, can fool the client into thinking its connected to the server, when its not actually, and results in the sort of weird connection errors you are seeing.
    – Tom
    Feb 25, 2012 at 3:36
  • Hmm. The ab -c5 -n1000 command worked great the first time. When I bumped the level of -c up it failed immediately, and now -c5 is getting the "connection forcibly closed" error.
    – Kevin
    Feb 25, 2012 at 5:40

1 Answer 1

Reset to default
3

Its likely that you would have to look at the MaxClients setting and the ServerLimit settings for your apache instance, which might be lower than you might expect to be sure its not an apache limit being imposed. However a quick look at the docs suggest that the default for both is 256, which is less that your ab imposed 50 connections limit.

Is the server actually crashing, ie you need to restart httpd or does it continue to serve requests once the load has been removed?

I would set loglevel debug and httpd.conf, and try with fewer connections and total n, and ramp up the test. I would also look and see what the values for RLimitCPU
RLimitNPROC
RLimitMEM

are as they are also limits at which apache might start killing processes/threads/connections etc.

Improve this answer
7
  • I added the following to my httpd.conf: <IfModule mpm_prefork_module> ServerLimit 1024 StartServers 10 MinSpareServers 10 MaxSpareServers 20 MaxClients 1024 MaxRequestsPerChild 0 </IfModule> It seems to have helped. I get through the test once in a while with mostly failures due to "Length". To answer your question, it continues serving requests shortly after it's self invoked restart. Checking the values right now will update.
    – Kevin
    Feb 25, 2012 at 2:33
  • Those three- RLimitCPU, RLimitNPROC, RLimitMEM aren't even in my httpd.conf so im assuming the default values are taking over.
    – Kevin
    Feb 25, 2012 at 2:37
  • I tried testing with a few values, now I don't even get past 150 requests. Any recommended settings?
    – Kevin
    Feb 25, 2012 at 2:50
  • I think start by running with the verbose option -v 3 you want to see the error that apache is returning.
    – Tom
    Feb 25, 2012 at 3:03
  • 1
    Well, you go the ball rolling and i'm grateful for that. I'll up vote your answer when I get 15 rep here. All my rep is on stackoverflow.com. It's very interesting information- especially now that i'm writing web services it has become even more important to understand Apache's internals.
    – Kevin
    Feb 28, 2012 at 21:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .