We're running a LAMP stack (Perl, MySQL) and have recently experienced roughly a tripling of our database load due to increased usage and shifting user behavior... which has overloaded our system.

Using a fairly standard 3-tier architecture, lots of webservers (doesn't seem to be the problem), a few specialty servers (don't think it's the problem) and a rather large MySQL database on modern Sun hardware with lota (72GB) of memory (the DB seems to be the current bottleneck).

Two questions I guess:

1) Anyone know of some resources they could point us to to help us work through our load issues? Buying hardware is definitely on the table; as is config changes...

2) Anyone out there with experience massively scaling a LAMP stack who would be interested in paid consulting work? We need someone immediately, preferably willing to travel to Raleigh NC to help on-site... First goal is to get out of the current crisis; second goal would be to help plan for another 10x usage increase over the next 2 years.

(As a sub-question to question #2: where would you go to look for such a consultant???)

Thanks in advance! -Steve

  • I don't have a particlar answer, but I do know that OmniTI omniti.com is good at scaling web applications. Have heard good things about them. – Dave Drager Sep 22 '09 at 17:46
  • Thanks for the responses! Very much appreciated. We are logging all requests and have identified a few badly optimized queries already which has helped; we'll definitely try for a thorough scrub of our SQL schema, possibly with some outside help. Thanks again! – Steve Sep 24 '09 at 15:19

The good news, if it is a DB issue, then many types of DB work can actually be done effectively via remote connection. If you're able to quickly set up a VPN into the server cluster, and a remote SSH / terminal account, then your scope of consultants broadens quickly.

I've never worked with Percona myself, but their MySQL scalability + performance book is really good, as is their work on Maatkit. Other consultants who give back to the community include Jeremy Cole @ Proven Scaling and the Open Query team.


Buying hardware is definitely on the table; as is config changes...

Those would be #2 and #3 on my list of candidates, #1 being the database design. Usually, when the database load suddenly balloons without a corresponding growth in usage, there's a O(n^m) issue somewhere. A full table scan in a join, an inefficient denormalization, a n+1 glitch, maybe lock contention while inserting data, stuff like that. Solutions could be as simple as adding a few well-chosen indexes, or as complex as a redesign of the data model.

I haven't worked with MySQL in years, but what i remember is: you can have it log all requests and their execution times. You should look at that log and find out what exactly causes the database to become the bottleneck.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy