6

I maintain a Gentoo server with a few services, including Apache. It's fairly low-end (2GB of RAM and a low-end CPU with 2 cores). My problem is that, despite my best efforts, an over-loaded Apache crashes the entire server. In fact, at this point I'm close to being convinced that Linux is a horrible operating system that isn't worth anyone's time looking for stability under load.

Things I tried:

  1. Adjusting oom_adj for the root Apache process (and thus all its children). That had close to no effect. When Apache was overloaded it would bring the system to a grind, as the system paged out everything else before it got to kill anything.
  2. Turning off swap. Didn't help, it would unload memory paged to binaries of processes and other files on /, thus causing the same effect.
  3. Putting it in a memory-limited cgroup (limited to 512 MB of RAM, 1/4th of the total). This "worked", at least in my own stress tests - except the server keeps crashing under load (basically stalling all other processes, inaccessible via SSH, etc.)
  4. Running it with idle I/O priority. This wasn't a very good idea in the end, because it just caused the system load to climb indefinitely (into the thousands) with almost no visible effect - until you tried to access an unbuffered part of the disk. This caused the task to freeze. (So much for good I/O scheduling, eh?)
  5. Limiting the number of concurrent connections to Apache. Setting the number too low caused web sites to become unresponsive due to most slots being occupied with long requests (file downloads).
  6. I tried various Apache MPMs without much success (prefork, event, itk).
  7. Switching from prefork/event+php-cgi+suphp to itk+mod_php. This improved performance, but didn't solve the actual problem.
  8. Switching I/O schedulers (cfq to deadline).

Just to stress this out: I don't care if Apache itself goes down under load, I just want the rest of my system to remain stable. Of course, having Apache recover quickly after a brief period of intensive load would be great to have, but one step at a time.

Right now I am mostly dumbfounded by how can humanity, in this day and age, design an operating system where such a seemingly simple task (don't allow one system component to crash the entire system) seems practically impossible - or at least, very hard to do.

Please don't suggest things like VMs or "BUY MORE RAM".


Some more information gathered with a friend's help: The processes hang when the cgroup oom killer is invoked. Here's the call trace:

[<ffffffff8104b94b>] ? prepare_to_wait+0x70/0x7b
[<ffffffff810a9c73>] mem_cgroup_handle_oom+0xdf/0x180
[<ffffffff810a9559>] ? memcg_oom_wake_function+0x0/0x6d
[<ffffffff810aa041>] __mem_cgroup_try_charge+0x32d/0x478
[<ffffffff810aac67>] mem_cgroup_charge_common+0x48/0x73
[<ffffffff81081c98>] ? __lru_cache_add+0x60/0x62
[<ffffffff810aadc3>] mem_cgroup_newpage_charge+0x3b/0x4a
[<ffffffff8108ec38>] handle_mm_fault+0x305/0x8cf
[<ffffffff813c6276>] ? schedule+0x6ae/0x6fb
[<ffffffff8101f568>] do_page_fault+0x214/0x22b
[<ffffffff813c7e1f>] page_fault+0x1f/0x30

At this point, the apache memory cgroup is practically deadlocked, and burning CPU in syscalls (all with the above call trace). This seems like a problem in the cgroup implementation...

10
  • Did you check the system logs for possible bugs/errors?
    – Khaled
    Dec 11, 2010 at 7:30
  • There's nothing interesting in the logs, except lots of requests coming to Apache. Dec 11, 2010 at 7:43
  • You give very little information of your site setup before diving into some technical details. Apache servers should run OK with 2G RAM unless the sites are really visited. Any system and any OS will fail it it gets more in than it is set to handle. What about Apache settings, databases, PHP settings etc. Do you have the site directly exposed to the net or are you behind a proxy/cache? Tell us more, and maybe some-one can connect the problems to answers.
    – ompap
    Dec 11, 2010 at 9:22
  • Hi, I think you misunderstood my question. I don't want to optimize Apache (for now), I just want to stop it from crashing the entire server when it's under load. Currently, a DOS attack (easily reproducible with ab) brings the entire server down - my question is how to keep other services, including SSH, responsive. Dec 11, 2010 at 10:30
  • Suphp isn't exactly a fast way to use php. What happens if you use php without suphp?
    – user9517
    Dec 11, 2010 at 14:03

6 Answers 6

7

I hate to say it, but you appear to be asking the wrong question.

It's not about stopping Apache from bringing down your server, it's about having your webserver serve more queries per second - enough so that you don't have a problem. A part of the answer to the reframed question is then limiting Apache so that it does not crash at high loads.

For the second part of that, Apache has some limits you can set - MaxClients being an important configuration. This limits how many children it's allowed to run. If you can take load off Apache for long-running processes (large files being downloaded for example), that's another slot in Apache to be able to serve PHP. If the file downloads have to be verified by the PHP layer, they can still do that, and pass back out to a more optimised webserver for the static content, such as with NginX sendfile

Meanwhile, forking Apache every on every single request for the slowest way to run PHP - as a CGI (whatever apache MPM you may be using) - is also having the machine spend large amounts of time not running your code. mod_php is significantly more optimised.

PHP can do huge amounts of traffic when Apache and the PHP layer are appropriately optimised. Yesterday, 11th Dec 2010, for example, the pair of PHP servers that I run did almost 19 Million hits in the 24hr period, and most of that in the 7am-8pm time-period.

There are plenty of other questions here, and articles elsewhere about optimising Apache and PHP, I think you need to read them first, before blaming Linux/Apache & PHP.

5
  • 1
    Hi. Thank you for the detailed answer... I'm aware that I'm not really taking the straight-forward approach to solving my problem. The thing is that I have other services running on the server, and I really, really want them to keep running even when Apache is bucking under load - regardless of its configuration. This should sound simple in theory, but practice has shown otherwise... hence my question. However, I may as well discuss a few points in your answer directly: Dec 12, 2010 at 18:15
  • 1) I don't understand how can I free up an Apache slot for long downloads without putting something in front of Apache. AFAIK even a proxy request eats up a slot? 2) I need to run PHP scripts under different users - so that means either an experimental MPM such as peruser or itk, or php-cgi + suPHP. itk works okay, but AFAIK peruser is unstable for production use. 3) I already have the bare minimum of modules, and each apache2 instance eats up at least 12 MB of memory - I don't see how I could bring it down further. Dec 12, 2010 at 18:23
  • Just to make it clear: I'm not blaming Apache or PHP, I know it's quite far from its optimal configuration. However, I do believe that Linux (the kernel itself) is very much to blame here, especially considering my recent discoveries (see bottom of OP). Thanks again. Dec 12, 2010 at 18:24
  • Any long running process that holds open a connection will help kill any forking Apache. Nginx as a proxy will happily serve a 1000 connections per second, for static content, even when proxying for more dynamic back to Apache. I served over 900K blog images/css/js yesterday and sent another 19+M calls to a pair of PHP/Apache serving machines (itself, plus a 2nd machine). Dec 12, 2010 at 19:13
  • @CyberShadow, you could use MPM Apache and mod_fcgid + suPHP. Jan 13, 2011 at 7:38
4

When you are dealing with a production apache server, you MUST have an average process size, especially with php, I'll recommend you to:

  • Check your process averages memory consumption
  • Adjust MaxClients to AVERAGE_MEMORY / RAM_DEDICATED_TO_APACHE

Where RAM_DEDICATED_TO_APACHE it must be another estimation of the TOTAL_RAM minus the ram that needs the rest of the machine (and be generous with the database if you are running one in the same machine).

I really recommend you to use Varnish, you can easily run 2 servers on different ports on the save machine, and route the static files to an specialized file (media) server (lighthttpd, nginx) or an apache instance with worker and no extra modules. And of course catch the static content with varnish.

Split the load is important because you will be using the same amount of ram to deliver any static file (which needs less than 1MB) if you don't do it.

If you really need to make sure to never consume all the ram, you can install a new cronjob running each 2 minutes (less or more as you consider) with the following line, adjusting the 50 to any amount of the lowest ram, and keep this number above 30 at least; you'll need some ram to stop the server.

vmstat -S M | tail -n 1 | awk 'BEGIN{ "date" | getline date }{if($4 + $6 < 50){ system("/etc/init.d/httpd stop"); system("/etc/init.d/httpd start"); print "Rebooting apache  on " date >> "/var/log/apache-reboots.log"}}'

This is a very hakish (dirty) way of limit you ram, but it can be very helpful when you are not really sure about your average memory per apache process, and if you see several reboots in you log file ("/var/log/apache-reboots.log"), the you should tune your apache MaxClients, MaxRequestsPerChild, ThreadsPerChild to avoid futures hard-reboots, with the time and tunning, you will have the exact configuration for your server.

2
  • Hmm, thanks for the advice, but: 1) it takes way less than 2 minutes to fill up memory with a full-fledged DOS/DDOS 2) stopping Apache "the nice way" takes quite a while when it's overloaded, I usually end up doing a killall. Anyway, the memory limitation was supposed to be done by cgroups, but as I've said I ran into other problems (what seems like I/O scheduler starvation) when using them. Dec 15, 2010 at 11:14
  • if it takes 2 minutes to fill up your memory from DOS attack, your problem may very well be a lack of optimization, insufficient configuration, or a lack of security. it should be able to ward of additional load from each client fairly quickly. after the first few requests, you should be able to detect the denial of service. if it is a distributed attack, you then have to protect your self by optimizing your server as you were for heavy load. I have NEVER been able to max out my memory, even if i DOS myself from a machine with unrestricted access to the http daemon.
    – RapidWebs
    Jul 7, 2014 at 10:59
1

A few general things you can try:

  • It is hard to tell from your description whether Apache/Linux is actually crashing or just becoming severely overloaded. I would suspect that you just have a server that has such a high load that the only effective course of action is cycling the power. I would approach the problem as an overloaded server unless there is specific evidence of an actual crash. If you optimize the server's performance but it still actually crashes then you can work to find and address that problem.
  • You generally never want your server to get in a condition of it regularly hitting the swap, especially any Apache instances. You can quickly get into a run away load situation where the server is running fine but when the traffic increases a few percent it starts to use the swap and the load sky rockets causing the site to be slow or inaccessible. To prevent Apache from use the swap reduce the number of max clients/connections and/or reduce the memory use by disabling any modules that are not needed. See also the next point.
  • You mention that connections in Apache are being used by long requests like file downloads. To help reduce this problem you can use a second web server (like lighttp) setup to serve just static content which Apache forwarding/redirecting requests to it. This frees up connections in Apache to do the heavy work and let you reduce the number of max clients/connections.
  • If you need to prevent a DoS, whether on purpose or accidental, there are various Apache modules you can install and setup. I've used mod_evasive and mod_limitipconn for example which worked well enough to prevent the less malicious types of DoS.
  • Don't dismiss optimizing Apache or other parts of the OS or application. Computers are very good at doing exactly what you tell them so if your Apache settings say "use more resources than this server has" then it will do exactly that. Like a lot of software, Apache is meant to work well with a huge range of hardware and applications but needs to be correctly setup for both. The default configuration only works well for a simple, low traffic web site.
  • With a little tuning you should be able to find a balance where the server gets to a high load but is still responsive enough to log in and check on it. At this point your options are either to profile and optimize the application, including adding caching layers, or get better hardware. This step should be after getting Apache setup correctly.
5
  • Hi. First of all, thank you for the detailed answer. I used "crash" a bit loosely, and yes - I meant becoming practically unusable due to swap or the I/O blocking issue. However, your answer avoids my real question - enforcing hard limits on Apache (or any service in general) so it can't crash the system, regardless of the service's configuration or implementation. Thank you for your advice, however I'd like to only proceed with optimizing Apache after I solve the problem, or convince myself it is truly unsolvable. I've done a bit more research today, so I'll add that to the question. Dec 12, 2010 at 17:15
  • Your "hard limits" for Apache are set in its configuration. For example, if you set MaxClients to 100 but your server's RAM only allows 75 then you're going to run into issues when you exceed 75 clients. I would start with conservative Apache settings that don't permit it to "crash" and from there work on other optimizations as needed (caching, static web server, application profiling, etc...).
    – uesp
    Dec 12, 2010 at 18:54
  • I'm sorry, but you're not suggesting anything that I haven't already tried or even not described in my original question. You're also missing the factual and intended meaning of "hard limits" - 75 long download requests will take a measly amount of resources, but 75 requests to a DB-heavy PHP page will bring the server to its knees. cgroups are supposed to allow imposing hard limits on any group of processes, however they seem to have other problems, as described in the OP. Dec 12, 2010 at 20:15
  • I have no experience with cgroup but I had a similar situation as yourself where 75% of the Apache clients were being used to download static content wasting a large chunk of memory. I switched to using lighttpd for all the static content and left Apache to handle the dynamic content. This makes it easier to tune Apache and lighttpd for their specific tasks but is not necessarily the only solution and may not work for your specific situation.
    – uesp
    Dec 12, 2010 at 21:56
  • Yep, it's definitely something to consider. The problem is that, in my situation, I need to serve many vhosts, and there isn't one of them that's generating all the load. On top of that, I have only one IPv4, so splitting static/dynamic content between two web servers is going to be very messy. Thanks anyway. Dec 13, 2010 at 8:45
1

Have you tried changing /proc/sys/vm/overcommit_memory to 2 ? This means the kenel will not allocate morememory than swap + a configurable percentage (proc/sys/vm/overcommit_ratio) of available ram.

In the case Apache will just fail as it't can't allocate the ram but services already loaded such as openSSH will continue to function.

I should add I have never tried this and just discovered this setting now. I would love to hear from any one who knows more. Otherwise I will test this tomorrow as I have exactly the same problem as described in the question.

2
  • Hmm, I've been told never to do this on a "production" system, as some system services may depend on overcommit to function. Also, it doesn't look like overcommit is cgroups-aware. Dec 13, 2010 at 8:32
  • you may want to have a read on these: etalabs.net/overcommit.html and ewontfix.com/3 and the discussion in the comments and come to your own conclusion. May 27, 2014 at 14:44
1

I found the problem...

Setting oom_adj to 15 for the whole memory-limited cgroup turned out to be very stupid. The adjusted score of all processes in the cgroup ended up all being 1000 - so when the cgroup ran out of memory, the system killed random processes and generally misbehaved.

I haven't had any system crashes after simply removing the line that set oom_adj.

1
  • Scratch that. I got sick of this problem and just wrote a cronjob script to kill Apache when it dies. It keeps getting swapped out because Apache eats all RAM regardless, but it kicks in sooner or later. Jun 8, 2011 at 10:15
0

this may be a little bit late, but I can say that blaming the OS is simply not the way to go. the OS is designed to meet the expectations of several different use case scenarios, therefore, you MUST configure it to meet your requirements.

not only this, but if you are having so much load that the system is crashing, then you have to optimize your system, or expand your network.

while over-optimizing too early can make things painful later on, not optimizing anything at all from the very beginning can have the very same consequences. it's all about balance.

however, you claim your goal is to prevent the system from crashing.. but then go on to say your solutions did not work. but some of them did work, you just were not happy with the results.

when you run out of memory, you swap. or things crash. end of story. if you don't want to swap, you have to:

a) limit your incoming connections. this has the effect of either turning people away

b) send them to the backlog. which has the effect of causing the site to slow, or die.

c) buy more memory. which you did not want to do

d) scale your network. which you also did not want to do

e) load balance. which is much like 'D'

without careful optimization, fine tuning, and expansion.. you cannot prevent all these things from happening.

in my experience, I learned that by using a granular mix of all the above generally caused things to work out in the end.

first off, I use apache2 + mpm_event + mod_fcgid. i'd carefully configure just about every possible option apache has to configure. this might take one evening to do, and another to get right. but it will be worth it.

I'd ensure that there is always one pool of workers ready to handle incoming connections, and let it grow, but cap this pool at some reasonable limit. this may sacrifice some speed, but results in stability.

second, I use both CGroups and IO Priority / CPU Priotiy to schedule different groups of services for different priorities.

anything that is 100% critical, which I always need access to, they are reserved a block of memory, and is set a higher IO and CPU priority. i'd whip up a script that sets these priorities every hour or so, so that children will inherit these priorities if their parent changes.

next is DNS, then Web, then Mail. in this order. this way, if something is misbehaving, more critical elements are favoured.

using monitor software, check if things are online, and if not, restart them. if anything has been using more than X MB of memory, for X Cycles... and you cannot connect to the service (i.e. on http://...:80) kill the service, and restart it. if it restarts more than X times in X cycles, time out (and notify for manual inspection). you might drop a few users occasionally, but atleast your system remains stable!

third, if you have a dedicated server, id put all website services on a separate disk. keep IO operations mainly over a different controller.

fourth, check out apache modules like mod_bw and mod_qos. mod_bw can do more than just limit bandwidth per virtualhost, and mod_qos... this is a quality of service module that can help mitigate some issues.

besides what you would expect from a full fledges QoS module, it can help with things like DoS preventing slowdos, limit NULL connections, and it can even turn off keepalive when the server reach a certain threshold of concurrent connections.

finally, I would set-up some intelligent caching front ends, or a load balancer. for example: using a few VM Instances, maybe use Varnish or NGinx, cache static files upstream. this will offload all the open slots Apache requires for serving that static content.

I'm really not sure what you expect to happen when you get alot of traffic. you want it to both remain stable, but you don't want to loose any functionality under stress, and you don't want to optimize anything, and you don't want to upgrade or extend your network?

well, if you don't want to CHANGE anything, how do you expect the problem to go away?

4
  • I think you misunderstood. My question was how to configure the system so that SSH remains responsive regardless of what Apache is doing - crashing, turning people away, queueing requests would all be acceptable as long as I can still SSH into the system. Such a simple thing one would expect of modern OSes, but apparently it is impossible. Jul 7, 2014 at 22:39
  • see the part of my answer that says "secondly". with proper optimization, prioritization, scheduling and affinity... I have never experienced being unable to SSH into my box. even under such high load that Apache begins to backlog my customers. if SSH is reserved a chunk of memory, if it has a higher CPU priority / IO priority. maybe a reserved CPU. you should be able to circumvent this. if apache is crashing your server, it may be a result of poor resource utilization. I'm not saying that you dont experience this, but it certainly is possible to achieve what you are trying to do.
    – RapidWebs
    Jul 7, 2014 at 23:35
  • I could be wrong, but maybe you are hitting some sort of bottle neck somewhere. perform a stress test from a machine in your local network, and double check what the IO for disk, network, CPU and memory looks like. if you can track down what SSH is waiting on during the denial of service, you should be able to optimize things to reserve enough resources, and allot them to the SSH daemon.
    – RapidWebs
    Jul 8, 2014 at 0:05
  • Thanks. I did try cgroups but I think I was running into problems due to a buggy kernel (ah, Gentoo...). SSH was waiting on I/O (things were being swapped out, often preemptively). Jul 8, 2014 at 5:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .