Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Today I (accidentally) ran some program on my Linux box that quickly used a lot of memory. My system froze, became unresponsive and thus I was unable to kill the offender.

How can I prevent this in the future? Can't it at least keep a responsive core or something running?

share|improve this question
add comment

2 Answers

I'll bet that the system didn't actually "freeze" (in the sense that the kernel hung), but rather was just very unresponsive. Chances are it was just swapping very hard, causing interactive performance and system throughput to drop like a stone.

You could turn off swap, but that just changes the problem from poor performance to OOM-killed processes (and all the fun that causes), along with decreased performance due to less available disk cache.

Alternately, you could use per-process resource limits (commonly referred to as rlimit and/or ulimit) to remove the possibility of a single process taking a ridiculous amount of memory and causing swapping, but that just pushes you into entertaining territory with processes that die at inconvenient moments because they wanted a little more memory than the system was willing to give them.

If you knew you were going to do something that was likely to cause massive memory usage, you could probably write a wrapper program that did an mlockall() and then exec'd your shell; that'd keep it in memory, and would be the closest thing to "keep a responsive core" you're likely to get (because it's not that the CPU is being overutilised that is the problem).

Personally, I subscribe to the "don't do stupid things" method of resource control. If you've got root, you can do all sorts of damage to a system, and so doing anything that you don't know the likely results of is a risky business.

share|improve this answer
add comment

This is something particularly difficult to prevent. It's because the kernel starts swapping. One solution is to turn swap off. When the system runs out of memory, rather than start swapping, the kernel will kill some processes; usually it picks up the correct process to kill, but it is anyway better to kill a random process than to have an unresponsive system.

This can be a particularly good solution for servers, because servers often have enough RAM and when they start to use swap space it means something's wrong anyway. However, desktops usually need the swap space, so I think there's no good solution for desktops. I often turn swap space off in servers, especially when there is suspicion of a memory leak.

share|improve this answer
3  
Turning off swap on any system is a bad idea, because it doesn't allow the unused pages to be swapped out and the free space used for disk cache. This is especially true when there's a memory leak. –  womble May 19 '12 at 10:55
1  
And with swap off, the system can still get slow due to paging. It will just be paging clean pages madly instead of dirty ones. (Since, without swap, it can never evict a dirty page, it will always have to evict clean ones.) –  David Schwartz May 19 '12 at 11:35
    
I have a server which has a memory leak. The first time it happened, I had to press the reset button, because the server became unresponsive. But now that I've turned swap off, the server just kills the apache child if it becomes too large (it's a safeguard in addition to MaxRequestsPerChild). The result is that the server runs without problem. It doesn't have many unused pages anyway, and it certainly isn't paging clean pages madly. –  Antonis Christofides May 19 '12 at 11:52
    
@AntonisChristofides: I'm not sure what you think the takeaway lesson from that is. Your solution is certainly a bad one because it hampers performance due to the inability to evict rarely-accessed dirty pages from physical memory, it didn't solve the underlying problem, and you run the risk that the OOM killer might kill a critical process. You happened not to encounter the particular hazard I was warning about, but you're still at risk for it because you have no swap. –  David Schwartz May 19 '12 at 13:33
    
Some general advice with memory problems of that kind: Leave swap on. look at what the slub/slab allocators are doing, they can get you into memory problems from behind. Check whether tuning "swappiness" and "vfs_cache_pressure" helps. –  rackandboneman May 19 '12 at 15:50
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.