So, once in a while, I see in firebug that a request takes over 15 or even 60 seconds to respond and sometimes never. Here is what I've ruled out:

  • It's not the CPU, cuz every time I check the Server load its less then 6 for all 3 numbers
  • It's not the memory, because that's fairly low too, less the 50%
  • It's not the I/O anymore, because I've seen the graphs that Joyent sent back to me when I requested them, and they show less then 3MB of I/O (mostly all read).
  • It's not the SQL performance - I've profiled every last SQL command that runs, and they're all (99.9% of them anyway) running in less then 30ms, most run in less then 5ms.
  • Oh and I've been profiling all the script execution times, and even the when the problem occurs, the script always manages to finish in 50ms or less (that's 1 / 20th of a second ). Now, I do run a lot of Ajax calls. 1 every 2 seconds per user and I have 300 DAU+. But, even if all 300 are playing simultaneously, that's still only 150 calls per second max. The only other thing I can think of is that one of my neighbors is funky. The problem is highly intermittent. 99% of the time it works perfectly and there's excellent performance. but 99%+ is not good enough.
  • Eventually the performance gets so bad I have to restart the server, at which point everything is fine again. I've done this about 4 times now. Any ideas? Note: this is on joyent, vps, intro package 256mb of ram with bursting.

here are the mysql dump info:

Traffic     ø per hour
Received    18 MiB  29 MiB
Sent    134 MiB 221 MiB
Total   151 MiB 251 MiB
Connections ø per hour  %
max. concurrent connections 5   --- ---
Failed attempts 0   0.00    0.00%
Aborted 0   0.00    0.00%
Total   9,418   15.59 k 100.00%
  • Whenever this happens, run a traceroute or mtr to your server. Could simply be network troubles somewhere - although it's more likely your server is just overtaxed - they'll cram as many VPS as possible on one server. You won't get 99% excellence unless you open your wallet bigtime
    – nos
    Mar 3 '11 at 2:25

Okay, I think I figured out the problem. The number MaxClients in Apache was set to 8, which is way too low of a value. I increased that, and it looks like things are better now.

  • 1
    Note that with only 256Mb of RAM running both Apache and MySQL you may run into swapping issues if you set MaxClients too high (8 may even be too high). Once your web server starts using swap memory your performance will plummet.
    – uesp
    Mar 5 '11 at 21:50
  • If this fixes the issue, you can mark this as the answer. Mar 5 '11 at 21:52

Try running ps -faux and then look at your process. Pay special attention to the STATS column. If you see "Ss" then your script is being blocked.

If its a database connection make sure you have wait_timeout in MySQL (statement_timeout for ya PostSQL users out there), set or else you'll potentially block instead of return when you are running a database query.

You can also run strace -p <process id> to see what system calls your process is running. Hope you're good at C :-)

I would add logging to your script to find out what is blocking that 1% of the time. Good luck.

  • The threads show that they are sleeping. Logging shows that the script finishes, lightning fast. (50ms or less), and yet it takes 15-60 seconds round trip. When the server is in this state, even simple html pages take over 10 seconds to return.
    – Pascal Aschwanden
    Mar 4 '11 at 1:01

If your database has performance issues it could be that you are not closing your connections. You can turn on connection pooling to eliminate that. You can also turn on output caching to make sure the queries come back quickly.

  • the total elapsed time of execution for script (as defined from the first line of code to the last line of code) is less then 50ms, so I don't think its a DB performance issue. It's either waiting before or after my script executes. but waiting for what?
    – Pascal Aschwanden
    Mar 3 '11 at 2:54
  • Well it could also be that it's waiting to connect. If you don't have connection pooling on, and you have a lot of connections to the database it could be waiting in line. You can also turn up the maximum number of connections to see if that's the cause as well.
    – Inturbidus
    Mar 3 '11 at 14:46
  • Or is it not running the script at all? What kind of script is it? PHP, Python??
    – Inturbidus
    Mar 3 '11 at 14:47
  • it can't be that because, the waiting time would show up in the lengthy script execution, since my mysql_connect call is after the profiling begins. The profiling begins on the very first line of code that executes.
    – Pascal Aschwanden
    Mar 3 '11 at 19:42
  • Also note: even very simple HTML web pages on that server go extremely slow when the server gets into that locked up state. These server crashes are happening almost every day. As soon as i reboot, everythings okay again for about 5 - 40 hours.
    – Pascal Aschwanden
    Mar 4 '11 at 3:51

Your server is simply underpowered. 150 requests per second is a lot to be sending to a VM which is running a full LAMP stack.

The behavior you are seeing is Apache being fully occupied by 8 concurrent requests, therefore refusing the 9th.

You've answered that you increased your MaxClients to 8 and that solved the problem. However you are now very likely to swap under load and cause your entire server to go down. I go into a lot of detail regarding properly configuring a LAMP stack in my answer here.

With the numbers you provided you simply cannot handle a load of 150 rps. With 150 rps and each execution taking 50ms, that's 7.5 seconds required to handle those 150 requests. This translates to requiring a minimum concurrency of 8. However, that 8 is the bare minimum insofar as it will only work if your 150 requests are more or less evenly distributed. If you get a burst of 50 requests in say a 10 ms period, well you're going to drop 42 of them.

You need to increase your MaxClients to be able to comfortably sustain whatever load peaks you expect. However there is literally no room to do this on a 256M server. I'd guess that your average httpd process size is anywhere from 10MB to 20MB. You can't just spin up 15 of these without swapping.

  • Why does anyone think that 256M on a full LAMP server is adequate? Maybe 5 years ago on static page websites, but with everything going to database driven PHP apps, it's pretty laughable. Jun 3 '12 at 16:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.