Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a VPS running apache that keeps running out of memory at Dreamhost. Dreamhost says that the problem is the wordpress configuration, but I think that's their stock answer. I've looked and the main web hosting is coming from mediawiki. I sometimes have 20-30 http processes running, and they are all running php, and mediawiki is powering my top two sites.

So I am looking for suggestions on ways to reduce the memory footprint of mediawiki. I'm currently running version 1.16.4, which I see is significantly behind the current version. (Dreamhost is supposed to upgrade it for me, but apparently they haven't.)

  • Does version 17.2 have lower footprint than 16.2?
  • Is there a clever way that I could use caching to reduce the amount of memory?
  • Are there configuration options that will reduce the memory?
  • Why do I have both apache httpd running and php5.cgi running?
  • Is there an easy way to find out which mediawiki parts are using the most ram?
  • Is there a way to reduce the number of files that are fetched? My web logs are filled with fetches to user.gif, bullet.gif, external.png, document.png --- how come mediawiki's themes don't use sprites?

Thanks!

share|improve this question
    
How much memory? –  Tim Jan 17 '12 at 13:39
6  
20-30 httpd processes will consume a lot of memory. Firstly determine if your problem is PHP, Apache, or MySQL. If you are memory limited, you may want to reduce the StartServers, MinSpareServers, MaxSpareServers, and MaxClients values in httpd.conf. Consider using an alternate FastCGI implementation (e.g. PHP-FPM); reduce the number of loaded Apache modules (consider using nginx or lighttpd if possible); also, you may be able to reduce the memory_limit in php.ini (although, that won't work with all scripts). –  cyberx86 Jan 17 '12 at 13:49
    
@cyberx86 That should be an answer. It's worth upvoting and a candidate for acceptance. –  Jeff Ferland Jan 17 '12 at 15:35
    
@JeffFerland: I am starting to think I have a real problem choosing between a comment and a answer. You'd think it was simple - comments as for clarification, answers answer the question. However, I seem to be of the misguided notion that an answer must be fairly complete and answer all parts of the question (so I tend to put suggestions and partial answers as comments). (My average comment length is 380 chars, my average post length is 1500 chars, and my comment to answer ratio is 3:2). I think I will take your suggestion, although I will need to improve my 'comment' a bit. –  cyberx86 Jan 17 '12 at 16:23

4 Answers 4

up vote 5 down vote accepted

My first suggestion would be to ensure you are fixing the right problem.

  • Track your memory usage over a reasonable time frame and see how high it goes (and if you can correlate this to something such as increased traffic).
    • If you already have some monitoring in place (e.g. Munin) you should be able to see memory trends
    • Otherwise, use sar (e.g. if you already have it setup, sar -r -f /var/log/sa/sa17 will give you today's memory information).

Determine what process(es) are actually using your memory.

  • Your problem might not be directly related to MediaWiki. While PHP might be consuming a lot of memory, MySQL and especially Apache are good candidates for significant memory usage.
    • Use top (or htop) or ps aux --sort -rss to see which processes are consuming the most memory.
    • If your problem is PHP, you may have some success reducing the memory_limit in php.ini

Reduce Apache's memory usage

  • 20 to 30 apache processes will consume a lot of memory (likely over 500MB)
  • If you can, switch from Apache to a lightweight web server such as nginx or lighttpd. These should work with most CMSes, although some configurations (e.g. using .htaccess files) are not supported.
  • Eliminate the apache extensions that you don't need - Apache will load an almost complete copy of itself - including all extensions, etc - into memory for each request it processes.
  • Reduce the number of server processes that Apache spawns. Apache processes usually start around 10MB each and with use can increase to 30+MB each.
    • If a moment of downtime is acceptable, consider the following approach (otherwise just estimate and do the math):
      • After a few hours of use, look at the average memory used by your Apache processes
      • Stop Apache and note your used memory - this should tell you how much your operating system, and all running services (MySQL, etc) need. Restart Apache
      • Take the difference between your total memory and that used by your base system, subtract a bit (10% at least) for safety, and divide by your average apache process size.
  • Set low values for StartServers, MinSpareServers, MaxSpareServers, and MaxClients. Keep MaxClients lower than the number you calculated above, and the other values lower still.
  • Set MaxRequestsPerChild to a non-zero value (100-300 should be good)
  • With fewer servers processes, you don't want any tied up for too long - so ensure your KeepAliveTimeout is low (10s should be adequate, possibly lower, no higher than 15s - the value depends on how your site is used)

There are additional suggestions on this very good guide to optimizing Apache for low memory.

Does version 17.2 have lower footprint than 16.2?

  • There are actually only 6 versions (16.2-16.5, 17.0-17.2) between these, moreover, the minor versions are usually security updates - so I wouldn't expect major changes, except perhaps for version 17.0 (and a quick look at the changelog doesn't suggest any major changes to memory management). If you really think it is the problem, launch a virtual machine (e.g. with VirtualBox), install the two versions, and run a load test (ab, siege, httperf, etc) on them - monitor memory usage and compare the results.

Is there a clever way that I could use caching to reduce the amount of memory?

  • This depends on what the source of your problem is:
    • If it is PHP generate static copies of your pages when they change, and serve those.
    • If your problem is with Apache though, serving static assets will still require a lot of memory (although, caching is always a good idea).
      • You could use a CDN to reduce requests to static assets - which should help with memory usage on Apache.

There are some less than ideal options you may consider:

  • Using a lightweight server as a reverse proxy - it will help with the static requests, and if those make up a large portion of your requests, should help with memory usage (after Apache is properly tuned) - however, running an additional server uses some additional memory (and adds complexity to the system).
  • Use a caching layer such as Varnish - usually this is intended to run from memory - with the intention of serving pages faster at the cost of using more memory - you can, however, set it up to use a file as the cache. Much like using a reverse proxy this will reduce the load on the backend, but will itself require some memory - if you are up for experimenting you can see if the gains offset the cost.
  • Verify that your opcode cache (e.g. APC) is working, and possibly use a file backed store instead of memory for storing the cache.

Why do I have both apache httpd running and php5.cgi running?

  • Likely because you are using FastCGI. The requests for PHP files are not executed by Apache (as would be the case with mod_php) but rather, by the CGI interface of PHP. You may find that another CGI interface - PHP-FPM - offers better resource management (it can be used with mod_fastcgi).

Is there an easy way to find out which mediawiki parts are using the most ram?

  • I'd suggest that the best way to accomplish this is to disable whatever you can (extensions/plugins, etc) and run a load test. You may have a bit of success with some profilers (e.g. XDebug), but I don't expect that the results will be that easy to act on (and usually tend to be more in the form of time spent). If your requests are taking a long time to execute, some process managers (e.g. PHP-FPM) offer a 'slowlog' functionality.

Is there a way to reduce the number of files that are fetched? My web logs are filled with fetches to user.gif, bullet.gif, external.png, document.png --- how come mediawiki's themes don't use sprites?

  • You could look into Google's mod_pagespeed - it will help you out with minification, optimizing images, etc - although, it does take some effort to setup properly. Beyond that, you can modify themes to your liking or use another theme. Ensure that images, etc are cached by your user's browsers. Possibly reduce logging for certain types of assets (e.g. static objects)
share|improve this answer
    
Thanks for all the good ideas. –  vy32 Jan 17 '12 at 23:38

If possible, switch from Apache to lighttpd or nginx. Those can serve your static content very effectively. Then configure something like FastCGI or fcgid for your dynamic content. That way you can effectively separate the static and dynamic content and isolate those from each other.

And yes, I purposely left out the web links for each keyword I used; Google them and decide what fits your needs.

share|improve this answer
    
I'm already using FastCGI and I'm making significant use of htaccess files on the server for rewrites and access control. But I guess I could use it for the statics if I run it on a different port. –  vy32 Jan 17 '12 at 17:21
    
@vy32 Don't use htaccess. –  Shane Madden Jan 17 '12 at 20:03

I can confirm the high memory use of MediaWiki - but only as far as compiled PHP bytecode is concerned. The APC cache viewer reveals that the compiled bytecode amounts to roughly 20 MB after just a few hits. The bytecode for all "normal" sites (not using WordPress, Drupal or Joomla) are less than 1 MB.

Then the real memory use reported by PHP: A quick test revealed that a MediaWiki page load only consumes about 5 MB of memory which is pretty much the same as that of Joomla. Some more lighweight content management systems use considerably less: some 1.8-3 MB per page load (of course depending on the complexity of the page structure).

As for bad memory use I have seen, though, that a standard WordPress installation and an utterly simple page may use up to 20-30 MB memory, with more complex page structures much more and quite a bit of CPU time. So DreamHost really knows the most common problem.

share|improve this answer

I would suggest you to migrating away from the CGI/FastCGI in favour of PHP-FPM + nginx. PHP-FPM keeps a pool of PHP interpreters spawned and doesn't spawn every single interpreter per request (which I think even FastCGI does).

From my own tests I've seen memory consumption drop by min 15% up to 35% in combination with a well configured bytecode optimizer (I use APC with good results). The real deal was getting away from Apache and FastCGI as the apache workers pool was spawning too many childs and each of them in turn was spawning PHP interpreters, which just after 4/5 days lead to out of memory / swapping with poor performance.

As suggested above analyze what is eating your memory right now, I bet APC is causing that as per this post

share|improve this answer
    
Thanks for the suggestion. –  vy32 Jan 3 '13 at 2:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.