Where to start...

I have a number of websites. I'm a developer. I'm not a very good server admin nor the "performance tuning" guy. I'm quite convinced that I've written somewhat OK code, both PHP-wise and MySql query wise.

One website in particular is now having some problems with high load from time to time, during peak hours. The stats are about 3,000,000 pageviews / month and there are pages that are quite devoid of logic but do carry some extensive media to download. It sits on a VPS on the Virtuozzo technique with 1,5 G of RAM available (but never uses more than half of that, peak or no peak).

My problem is that I don't really know where to start in finding out what I can do to "help" my site. I don't know what stats I should be looking at to figure out what is reallyp causing the load. Is it mysql that starts using too much tmp-disk? I don't know. Is it some of my PHP-code that is expensive to execute? I don't know. Is it Apache that spawns too many threads? Couldn't tell.

I am totally in the dark here on how to attack this problem and need some guidance please.

Should I install Munin? What do I look for in Munin then, when I do get the stats.

Looking at TOP in peak hours I can see that the load sometimes hit as high as 20, when off peak hours it rarely goes above 1. I can see Apache thread count at about 100, tcp connections at about 300, mysql threads only 2 (mysld and mysqld-safe, whatever that is).

I have an op-cache installed already (eAccellerator) but I have no real stats to give you there other than it seems to work OK since the load of the pages goes quite fast.

Well, any help at all in where to start and how to attack this is greatly appreciated.

EDIT Peak stats: about 250 req / min, I'm on a ServInt VPS and they say all machines have 4 CPU's but I guess that's only for burst (if even) so I couldn't say for sure about the CPU I'm sorry to say.

  • Can you give the rought stats of the site during peak hours - req / minute? What cpu spec is the VPS (to put a load of 20 into perspective - if it's a single core then this is v. bad) – Ian Nov 6 '09 at 9:00
  • @iAn: See my edit above. – Jeff Nov 6 '09 at 9:21

I've written a couple of wiki pages for our company knowledge base that might be of assistance:

In all honesty, you really don't want to be dealing with this stuff yourself -- you're a developer, focus on your strengths. Server performance and tuning is a whole other world to development. I'd recommend finding a good sysadmin (or team thereof) to deal with these sorts of problems. At the risk of getting a bit spammy... the company I work for does a lot of this sort of work...

  • Well, first of all thanks for the reply. You're of course correct about the best thing being to hire someone to look into this. however, being on a budget I wanted to see if it's at all possible to learn about this myself to at least let me know what questions are relevant. Thanks. – Jeff Nov 6 '09 at 10:12
  • Good couple of articles. – RainyRat Nov 6 '09 at 10:30
  • Yeah, you can learn it yourself -- sysadmins aren't a separate species. It's just a matter of whether you want to commit the time -- years, to do it properly -- to basically learning a whole new skillset. – womble Nov 6 '09 at 10:57

I'm on a ServInt VPS and they say all machines have 4 CPU

Usually that means all host machines have quad cpus. They don't tell you how many VPS clients are running on those servers though. It could be as little as 4, it could be as many as 100. (you can guess which end of the range you're on by looking at your monthly bill). If you're sharing the server with a lot of other people, I'd expect peak time to slow you down considerably more than you'd expect from more people using your site. A server load of 20 does suggest that might be the case.

To detect performance issues, take a bit out of one of the systems that make up your site: it could be apache networking, too many users for example; it could be un-performant PHP code running too many loops; it could be badly designed queries, not using sql indexes or retrieving too much data. Choose 1 thing to look at performance logging for that component. eg. for SQL, turn on the slow query log, check the buffer size parameters, check the number of connections. Then go to the next component and check that. Repeat until happy.


Jeff, as a php developer one thing that you can definitively answer is whether or not it's your code that is the bottleneck. Load averages is CPU, so if you're hitting 20 at peak times you're getting slammed. usually thats not an indicator of apache using up too much cpu, its typically your code, or your queries.

The best tool that I've found for digging right into what your code is doing is XHProf. XHProf is the PHP profiling tool developed by facebook that will allow you to see exactly what is happening in your code. It shows you the cpu time, wall time (typically waiting on db), the function calls, the call stack etc. It can make an expensive loop in your code quite obvious.

You can even run it against your code multiple times, and aggregate all the results together to give you a clearer picture of whats happening. Maybe 95% of your site is lightning fast, but there's just a couple of resource heavy urls that when hit, bog down the rest of the system. Heres a good link for getting it set up: http://techportal.ibuildings.com/2009/12/01/profiling-with-xhprof/

Another thing to look at is if you have an opcode cache installed on your server - XCache, eAccelerator, or APC. When the php interpreter is fired up on every request, its very CPU intensive to turn your PHP code into opcodes every time. An opcode cache will interpret your code once and re-use that under the assumption that your code isnt changing often enough to warrant recompiling it every time.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy