1

This is not the typical serverfault question, but I'm out of ideas and don't know where else to go. If there are better places to ask this, just point me there in the comments. Thanks.


Situation

We have this web application that uses Zend Framework, so runs in PHP on an Apache web server. We use MySQL for data storage and memcached for object caching.

The application has a very unique usage and load pattern. It is a mobile web application where every full hour a cronjob looks through the database for users that have some information waiting or action to do and sends this information to a (external) notification server, that pushes these notifications to them. After the users get these notifications, the go to the app and use it, mostly for a very short time. An hour later, same thing happens.

Problem

In the last few weeks usage of the application really started to grow. In the last few days we encountered very high load and doubling of application response times during and after the sending of these notifications (so basically every hour). The server doesn't crash or stop responding to requests, it just gets slower and slower and often takes 20 minutes to recover - until the same thing starts again at the full hour.

We have extensive monitoring in place (New Relic, collectd) but I can't figure out what's wrong; I can't find the bottlekneck. That's where you come in:

Can you help me figure out what's wrong and maybe how to fix it?


Additional information

The server is a 16 core Intel Xeon (8 cores with hyperthreading, I think) and 12GB RAM running Ubuntu 10.04 (Linux 3.2.4-20120307 x86_64). Apache is 2.2.x and PHP is Version 5.3.2-1ubuntu4.11.

If any configuration information would help analyze the problem, just comment and I will add it.

Graphs

info

collectd

New Relic

(Sorry the graphs are gifs and not the same time period, but I think the most important info is in there)

2 Answers 2

2

Hello Unfortunately It is difficult to find an immediate solution to your problem especially without a sys admin. This said I think that you can at least solve (on the long run) the memory hungry apache -as it seems from your relic report - with a nginx + apache + php to scale up the speed of your app considerably see http://www.richweb.com/nginx or simply search on google /ask your sys admin. As you use Zend framework you might also consider Zend Server http://www.zend.com/en/products/server/

Sorry if these ideas seems a bit generic and do not solve your immediate problem but, over the long term, this could provide you with a good solution.

Another suggestion is that, if you application tends to scale in the future, you might consider testing the same setup you have on amazon EC2 http://aws.amazon.com/ec2/ This would give you several benefits like:

  1. Scalability: you can run one main instance and clone it to a second instance to run anytime there is a need (e.g. something is slow and you want to see if it is the same on the second instance) or better with two instances and a load balancing.
  2. Cloning: with a traditional server it is fairly complicate to clone everything and be up and running in minutes. With EC2 you have this advantage;
  3. Simplicity: with the new EC2 web based interface you will not need your sys admin to launch a new instance and test it with a new IP (that can be assigned on the fly).

This might sound as a random suggestion not addressing your problem but, from personal experience, sometimes having the possibility to scale becomes essential as your business grow.

For immediate assistance in your case only a sysadmin can help you. If you want to hire one you might consider https://www.odesk.com but again you need to select a reliable one with very good feedback. If you want just a consultancy without having the admin to enter your server I am sure that you can find several ones available at very reasonable rates (20/30$ an hour) to give your some feedback.

3
  • Thanks. nginx for static files seems like a good idea in general, but can you tell me from which graph exactly you see that Apache needs too much memory and this is a problem? I want to point our sysadmins on this. EC2 and other cloud solutions are an option, but add lots of complexity I hoped to be able to postpone. Especially as the server runs fine 90% of the time, but only on peak times and even then I don't really get why it slows down and what exactly is the bottleneck.
    – janpio
    Apr 15, 2012 at 11:35
  • 1
    I actually simply checked which consumes more memory from processes list. Again is difficult to give a direct advice without more information. Apache should be more than covered anyway by your 12 GB Ram but on peak time this might be the issue. You might also consider upgrading to MySQL 5.6 that is much much faster than 5.5.
    – devnull
    Apr 15, 2012 at 18:53
  • Thanks! I think our sysadmins aren't ready for 5.6 yet, at least I haven't heard them using it anywhere.
    – janpio
    Apr 15, 2012 at 20:54
1

Serverfault, in general, is not your outsourced IT consultancy. We are setup to answer technical questions that are both specific (in that they can be answered) but also general (in that the answer provided could be of use to someone who has the same problem in the future, and might search the internet for their problem and will find your question and the answers will help them). Your question, unfortunately, fails on both counts.

I'll give you a certain amount of credit for at least providing a useful chunk of diagnostic information, which puts you in the top 1% of people who ask this sort of question. However, that doesn't change the fact that the question is basically "do my job for me", which is rather rude.

My only useful answer to you is to get in a consultant. My expectation, given your description of the problem, is that you'll end up requiring a rearchitecting of your application, possibly involving splitting out a DB read slave and using a separate machine for processing your async notifications. I'd also look at switching to a real-time work queue so that the app doesn't need to troll the entire database, or at the very least configuring some indexing in the DB and/or reworking queries to make the retrieval of that information more efficient. A competent consultant should be able to analyse your situation with performance metrics and an examination of the code and system operation and provide recommendations and assistance in implementing them. I work for a hosting company that provides all these services as part of our support package, but only for customers who are hosted with us, so I'm not available for those sorts of ad-hoc engagements myself (unless you want to switch your hosting...)

1
  • It's saturday night and my developer brain can't stop thinking about the thing my sysadmins couldn't figure out for me before the weekend, so I try to find help online. But yeah, I see and get your point. Are there any online communities where, based on the information I can provide right now, I can get help?
    – janpio
    Apr 15, 2012 at 1:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .