Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Posted this a couple days ago, have made some more progress. Thanks for the help, this is delaying our launch and I'm under a lot of pressure.

UPDATE: I don't think it's the server side, I think Chrome ect can't handle the number of changes that get sent when multiple ppl are making live updates at once.

Has anyone run into this? Know a work around?

Problem: Only about 3 or 4 users can use our server simultaneously before app grinds to a halt.

Currently we see massive spikes in the CPU usage with normal usage. This is easier to reproduce with real users than with automated scripts, for unknown reasons, but it's possible the scripts don't do a great job of simulating real usage.

Our architecture is as follows:

  • App server (Tornado) - single threaded, with an asynchronous IO Loop. We use Tornado to handle persistent connections associated with long-polling, as well as basic web requests.
  • Django ORM is used to interact with database, although most SQL is hand-coded MySQL database
  • Nginx serves static media and proxies other requests to Tornado
  • Everything is currently setup to run on one "Small" EC2 instance. Separating the servers between machines doesn't have a noticeable impact on performance
  • See EC2 server spec: http://aws.amazon.com/ec2/instance-types/ for more details on server configuration.

Note: All-in-all this isn't the ideal & most scaleable setup, but it should be able to handle more than 3 users!

Running top & viewing logs reveals the following:

  • CPU spikes are attributed mostly to Tornado, about 25% extra CPU usage per active user Low "steal-time", so our CPU power isn't being heavily throttled by EC2 (anymore)
  • DB queries are all between 0-200ms when CPU isn't spiking, but often 3 seconds or more during spikes
  • Memory usage is low and never spikes

Some things that have been tried to no avail:

  • Configure MySQL buffer sizes, indexes, etc.. I'm 99% sure this isn't a garden-variety SQL/DB optimization issue
  • Improve query times and reduce number of queries in all sorts of ways
  • Put servers on separate ec2 instances
  • Proxy between multiple app servers (this would obviously be a lot more scaleable, but it doesn't fix the 3-users-per-instance issue)
  • Upgrade the EC2 instance. Upgrading from "Micro" did help (due to CPU throttling issues) but only slightly increased our capacity
  • Deploy on non-EC2 server (Slicehost) - same problems
  • All servers have been load tested individually with simple test cases, and all have been able to handle 1000's of simultaneous connections
share|improve this question
1  
Have you put monitoring in place to 'see' what your server load/io/memory/interrupts/bandwitdh etc. are doing ? –  Iain Jul 18 '11 at 19:04
    
Run Wireshark on a Linux client and use the app and see how much actual traffic is being generated. Run an iptrace on the server and observe the same stuff. –  EightBitTony Jul 18 '11 at 19:35
    
Have you used developer tools to monitor your ajax requests? stackoverflow.com/questions/1820927/… –  sreimer Jul 18 '11 at 22:09
    
This looks like a repeat question: serverfault.com/questions/290845/… –  Mnebuerquo May 27 at 1:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.