Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

While stress-testing LVM on one of our Debian servers, I came across this issue where memory would fill up a lot to the point where it would run the server out of memory, but no process would claim the memory. See http://i.imgur.com/cLn5ZHS.png, and see http://serverfault.com/a/449102/125894 for an explanation on the colors used in htop.

Why is this happening?
And is there any way to see what process is using the memory?
Htop is configured not to hide any processes, so what is it that htop is missing?

In this particular case, I can fairly certainly say that it is caused, directly or indirectly, by lvmcreate, lvmremove or dmsetup, as I was stress-testing that. Do note that this question is not about solving the LVM problem, but about why the memory isn't claimed by any process. Stopping all LVM commands does bring the memory back down to <600MB.

Output of free -m:

             total       used       free     shared    buffers     cached
Mem:         32153      31958        194          0         52       3830
-/+ buffers/cache:      28075       4077
Swap:          975          0        975


Output of top, on pastebin due to the length of it: http://pastebin.com/WchrpF7W

share|improve this question
    
I'd be interested by a free -m and a regular top. –  user2299634 Jul 3 '13 at 15:53
    
@user2299634 I have reproduced the situation and added the output of free -m and top, as requested. –  SharkWipf Jul 4 '13 at 13:35
    
What is the output from /proc/meminfo ? –  Nehal Dattani Jul 4 '13 at 14:27
    
What tool are you using to do your stress test ? Do you have customs sysctl relative to memory management, buffers, storage ? Anything like bcache ou dm-cache ? –  user2299634 Jul 5 '13 at 16:19
    
I do not have access to the server for 2 weeks, so I cannot reproduce anything for a while. Nothing special in sysctl afaik, will doublecheck when I have access. The "stress test" is basically a bash loop creating/deleting snapshots, while a dd if=/dev/urandom is constantly changing the lv that's being snapshotted. The dd+snapshot is what makes the RAM fill up, which makes sense, but it not showing does not make sense. –  SharkWipf Jul 5 '13 at 20:08

1 Answer 1

Since you seem to be running I/O related tests (you didn't explain with details what you did with the LVM volumes), is there any chance that the behavior you noticed is related to the natural OS page cache?

I'm thinking about the Linux standard behavior of using (sometimes almost all) available memory (that is, memory that is not reserved by applications) to cache disk regions that were recently accessed, in order to speed up IOs.

Of course, active applications have priority over this cache, so in case a process needs more ram, the cache is dynamically shrinked.

There is a lot of documentation about Linux page cache on the web, here is one.

share|improve this answer
    
That was my first thought as well, but no, I am certain it is not. As you said, cache is automatically shinked when memory is needed. In this case, the server effectively runs out of memory. Besides, page cache shows as orange in htop. Thanks for your answer though. –  SharkWipf Jul 3 '13 at 13:57

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.