1

We're running a LAMP stack (Perl, MySQL) and have recently experienced roughly a tripling of our database load due to increased usage and shifting user behavior... which has overloaded our system.

Using a fairly standard 3-tier architecture, lots of webservers (doesn't seem to be the problem), a few specialty servers (don't think it's the problem) and a rather large MySQL database on modern Sun hardware with lota (72GB) of memory (the DB seems to be the current bottleneck).

Two questions I guess:

1) Anyone know of some resources they could point us to to help us work through our load issues? Buying hardware is definitely on the table; as is config changes...

2) Anyone out there with experience massively scaling a LAMP stack who would be interested in paid consulting work? We need someone immediately, preferably willing to travel to Raleigh NC to help on-site... First goal is to get out of the current crisis; second goal would be to help plan for another 10x usage increase over the next 2 years.

(As a sub-question to question #2: where would you go to look for such a consultant???)

Thanks in advance! -Steve

2
  • I don't have a particlar answer, but I do know that OmniTI omniti.com is good at scaling web applications. Have heard good things about them. Sep 22, 2009 at 17:46
  • Thanks for the responses! Very much appreciated. We are logging all requests and have identified a few badly optimized queries already which has helped; we'll definitely try for a thorough scrub of our SQL schema, possibly with some outside help. Thanks again!
    – Steve
    Sep 24, 2009 at 15:19

2 Answers 2

0

The good news, if it is a DB issue, then many types of DB work can actually be done effectively via remote connection. If you're able to quickly set up a VPN into the server cluster, and a remote SSH / terminal account, then your scope of consultants broadens quickly.

I've never worked with Percona myself, but their MySQL scalability + performance book is really good, as is their work on Maatkit. Other consultants who give back to the community include Jeremy Cole @ Proven Scaling and the Open Query team.

1

Buying hardware is definitely on the table; as is config changes...

Those would be #2 and #3 on my list of candidates, #1 being the database design. Usually, when the database load suddenly balloons without a corresponding growth in usage, there's a O(n^m) issue somewhere. A full table scan in a join, an inefficient denormalization, a n+1 glitch, maybe lock contention while inserting data, stuff like that. Solutions could be as simple as adding a few well-chosen indexes, or as complex as a redesign of the data model.

I haven't worked with MySQL in years, but what i remember is: you can have it log all requests and their execution times. You should look at that log and find out what exactly causes the database to become the bottleneck.

You must log in to answer this question.