Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a Windows Server 2003 machine running Apache2.2. Most of the time there is no load on the server, but we have a notification program on 3400 PC's that can request a small web page that plays a 64KB .wav file. When an event occurs those 3400 PC's all request the web page over the course of 3 minutes. On a few machine we saw the browser sit in the "connecting" state for a little over a minute before the page painted. What is happening, and how can we speed this up?

share|improve this question
    
Have you ran any perfmon counters to see what the issue is (CPU bound, disk i/o, RAM, etc)? –  tegbains Apr 30 '10 at 4:29
    
We were watching performance monitor on the server when the event occurred. The CPU and RAM usage barely moved. Disk queue times looked good too. I think the issue is with the number of simultaneous connections to the web server. –  Jason Lamoreux Apr 30 '10 at 13:10
    
There were a lot of good answers and useful comments, so thanks guys. The biggest problem ended up being that fact ThreadLimit does nothing when you are running Apache on Windows. The biggest help was bumping ThreadsPerChild up to 1024. We also enabled caching, and it seemed to speed things up even more. LoadModule cache_module modules/mod_cache.so <IfModule mod_cache.c> LoadModule mem_cache_module modules/mod_mem_cache.so <IfModule mod_mem_cache.c> CacheEnable mem / MCacheSize 4096 MCacheMaxObjectCount 100 MCacheMinObjectSize 1 MCacheMaxObjectSize 2048 </IfModule> –  Jason Lamoreux Aug 18 '10 at 18:03
add comment

5 Answers

If your ThreadLimit is 64 (ThreadLimit is a global limit) then, iirc, you're only going to be able to serve 64 clients at once. Figure it takes at least 500-1500 ms to serve each client, you've got a math problem.

If you can serve a single client in 500 ms, that's 128 clients served per second, that's 27.3 seconds to serve 3500 clients.

If you can serve a single client in 1.5 s, thats 42.66 clients per second, that's 82 seconds to serve 3500 clients.

2 seconds per client is 54.6 seconds to serve 3500 clients.

500 ms is probably reasonable figuring in TCP setup time, request time, transfer time, etc, depending on the latency in the network.

I'd run a test to see how long it takes to serve a single request at a time then do the math to figure out how long it should take to serve everybody if every request has that same serve time.

You probably need a total of 1000 ready threads to be able to serve that file in 3.5 seconds.

Also, you may want to see how long it takes apache to spin up threads - if it only keeps 30 threads when idle, and the machine is idle up until this point, then it has to spin up all those threads to be able to start serving them. Make sure your minimum number of idle threads is high.

Edit:

Hmm, I did some testing, and it looks like you can expect max 40 ms transfer times on something like a corporate network. If it takes 40 ms to serve a client, that's 1600 clients a second, a little under 2 seconds to serve 3500 clients.

The next best answer I have is that the rush of traffic is triggering ethernet faults. We used to have a web server that would work fine during the day but would turn to crap at night. Found out our backups were triggering faults on the switch uplinks, slowing the entire switch down. Bad autonegotiation + bad cables were causing it.

share|improve this answer
add comment

You have a fixed number of clients you can serve at one point in time. Each slot is used for the period it takes to transfer the file to your client. If you have 100 slots, and it takes the network 5 seconds to transfer the file (perhaps these computers are behind slow links, dialup), then you're going to need 175 seconds to serve all of those hosts. You'll look at the webserver and everything will look alright - not a lot of CPU load, not a lot of disk load, not a lot of network load. But those slow clients are killing you.

share|improve this answer
    
This could play into it, but all the computers are on the LAN, so the transfer should be pretty fast. –  Jason Lamoreux Apr 30 '10 at 13:15
add comment

It takes that long because you're performing a denial of service attack on your poor server.

Why do the computers need to fetch the audio file from a remote server if the alert is occurring locally? Couldn't they have a local copy distributed with the notification app?

share|improve this answer
    
The .wav file is generated by a text to speech engine on the server. It can change every time. –  Jason Lamoreux Apr 30 '10 at 13:07
add comment

There are few scenarios that so clearly show advantage of a good light webserver, especially a single-threaded one. Try either nginx or lighttpd, i'm sure either can comply with a much smaller machine.

share|improve this answer
2  
Too bad setting up nginx and lighttpd on windows is a huge hassle. –  Frew Apr 29 '10 at 20:20
add comment

I would suspect you need to adjust the values like threadsperchild to a much larger value in order to handle that load. The value controls the maximum number of requests that can be processed at once.

share|improve this answer
    
These are my current settings: ThreadLimit 64 ThreadsPerChild 64 MaxRequestsPerChild 10000 MaxMemFree 51200 Do you have a suggestion for an appropriate threadsperchild? We have 8Gig of RAM and Dual quad core CPUs. Even under load, the memory and CPU usage didn't even budge. –  Jason Lamoreux Apr 29 '10 at 17:53
1  
It would depend a lot upon your bandwidth and how fast those 3400 computers can retrieve the file. I would be tempted to experiment a bit and see what happens. Maybe go from 64 -> 256 -> 1024. –  Zoredache Apr 29 '10 at 21:29
    
I will give this a shot. How useful do you think the ApacheBench tool would be for testing this? –  Jason Lamoreux Apr 30 '10 at 13:13
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.