2

We have a Windows Server 2003 machine running Apache2.2. Most of the time there is no load on the server, but we have a notification program on 3400 PC's that can request a small web page that plays a 64KB .wav file. When an event occurs those 3400 PC's all request the web page over the course of 3 minutes. On a few machine we saw the browser sit in the "connecting" state for a little over a minute before the page painted. What is happening, and how can we speed this up?

3
  • Have you ran any perfmon counters to see what the issue is (CPU bound, disk i/o, RAM, etc)?
    – tegbains
    Apr 30, 2010 at 4:29
  • We were watching performance monitor on the server when the event occurred. The CPU and RAM usage barely moved. Disk queue times looked good too. I think the issue is with the number of simultaneous connections to the web server. Apr 30, 2010 at 13:10
  • There were a lot of good answers and useful comments, so thanks guys. The biggest problem ended up being that fact ThreadLimit does nothing when you are running Apache on Windows. The biggest help was bumping ThreadsPerChild up to 1024. We also enabled caching, and it seemed to speed things up even more. LoadModule cache_module modules/mod_cache.so <IfModule mod_cache.c> LoadModule mem_cache_module modules/mod_mem_cache.so <IfModule mod_mem_cache.c> CacheEnable mem / MCacheSize 4096 MCacheMaxObjectCount 100 MCacheMinObjectSize 1 MCacheMaxObjectSize 2048 </IfModule> Aug 18, 2010 at 18:03

5 Answers 5

2

It takes that long because you're performing a denial of service attack on your poor server.

Why do the computers need to fetch the audio file from a remote server if the alert is occurring locally? Couldn't they have a local copy distributed with the notification app?

1
  • The .wav file is generated by a text to speech engine on the server. It can change every time. Apr 30, 2010 at 13:07
1

I would suspect you need to adjust the values like threadsperchild to a much larger value in order to handle that load. The value controls the maximum number of requests that can be processed at once.

3
  • These are my current settings: ThreadLimit 64 ThreadsPerChild 64 MaxRequestsPerChild 10000 MaxMemFree 51200 Do you have a suggestion for an appropriate threadsperchild? We have 8Gig of RAM and Dual quad core CPUs. Even under load, the memory and CPU usage didn't even budge. Apr 29, 2010 at 17:53
  • 1
    It would depend a lot upon your bandwidth and how fast those 3400 computers can retrieve the file. I would be tempted to experiment a bit and see what happens. Maybe go from 64 -> 256 -> 1024.
    – Zoredache
    Apr 29, 2010 at 21:29
  • I will give this a shot. How useful do you think the ApacheBench tool would be for testing this? Apr 30, 2010 at 13:13
1

There are few scenarios that so clearly show advantage of a good light webserver, especially a single-threaded one. Try either nginx or lighttpd, i'm sure either can comply with a much smaller machine.

1
  • 2
    Too bad setting up nginx and lighttpd on windows is a huge hassle. Apr 29, 2010 at 20:20
0

You have a fixed number of clients you can serve at one point in time. Each slot is used for the period it takes to transfer the file to your client. If you have 100 slots, and it takes the network 5 seconds to transfer the file (perhaps these computers are behind slow links, dialup), then you're going to need 175 seconds to serve all of those hosts. You'll look at the webserver and everything will look alright - not a lot of CPU load, not a lot of disk load, not a lot of network load. But those slow clients are killing you.

1
  • This could play into it, but all the computers are on the LAN, so the transfer should be pretty fast. Apr 30, 2010 at 13:15
0

If your ThreadLimit is 64 (ThreadLimit is a global limit) then, iirc, you're only going to be able to serve 64 clients at once. Figure it takes at least 500-1500 ms to serve each client, you've got a math problem.

If you can serve a single client in 500 ms, that's 128 clients served per second, that's 27.3 seconds to serve 3500 clients.

If you can serve a single client in 1.5 s, thats 42.66 clients per second, that's 82 seconds to serve 3500 clients.

2 seconds per client is 54.6 seconds to serve 3500 clients.

500 ms is probably reasonable figuring in TCP setup time, request time, transfer time, etc, depending on the latency in the network.

I'd run a test to see how long it takes to serve a single request at a time then do the math to figure out how long it should take to serve everybody if every request has that same serve time.

You probably need a total of 1000 ready threads to be able to serve that file in 3.5 seconds.

Also, you may want to see how long it takes apache to spin up threads - if it only keeps 30 threads when idle, and the machine is idle up until this point, then it has to spin up all those threads to be able to start serving them. Make sure your minimum number of idle threads is high.

Edit:

Hmm, I did some testing, and it looks like you can expect max 40 ms transfer times on something like a corporate network. If it takes 40 ms to serve a client, that's 1600 clients a second, a little under 2 seconds to serve 3500 clients.

The next best answer I have is that the rush of traffic is triggering ethernet faults. We used to have a web server that would work fine during the day but would turn to crap at night. Found out our backups were triggering faults on the switch uplinks, slowing the entire switch down. Bad autonegotiation + bad cables were causing it.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .