8

While stress-testing LVM on one of our Debian servers, I came across this issue where memory would fill up a lot to the point where it would run the server out of memory, but no process would claim the memory. See https://i.stack.imgur.com/ismcH.png, and see https://serverfault.com/a/449102/125894 for an explanation on the colors used in htop.

Why is this happening?
And is there any way to see what process is using the memory?
Htop is configured not to hide any processes, so what is it that htop is missing?

In this particular case, I can fairly certainly say that it is caused, directly or indirectly, by lvmcreate, lvmremove or dmsetup, as I was stress-testing that. Do note that this question is not about solving the LVM problem, but about why the memory isn't claimed by any process. Stopping all LVM commands does bring the memory back down to <600MB.

Output of free -m:

             total       used       free     shared    buffers     cached
Mem:         32153      31958        194          0         52       3830
-/+ buffers/cache:      28075       4077
Swap:          975          0        975


Output of top, on pastebin due to the length of it: http://pastebin.com/WchrpF7W

5
  • I'd be interested by a free -m and a regular top. Jul 3, 2013 at 15:53
  • @user2299634 I have reproduced the situation and added the output of free -m and top, as requested.
    – SharkWipf
    Jul 4, 2013 at 13:35
  • What is the output from /proc/meminfo ? Jul 4, 2013 at 14:27
  • What tool are you using to do your stress test ? Do you have customs sysctl relative to memory management, buffers, storage ? Anything like bcache ou dm-cache ? Jul 5, 2013 at 16:19
  • I do not have access to the server for 2 weeks, so I cannot reproduce anything for a while. Nothing special in sysctl afaik, will doublecheck when I have access. The "stress test" is basically a bash loop creating/deleting snapshots, while a dd if=/dev/urandom is constantly changing the lv that's being snapshotted. The dd+snapshot is what makes the RAM fill up, which makes sense, but it not showing does not make sense.
    – SharkWipf
    Jul 5, 2013 at 20:08

3 Answers 3

3

Since you seem to be running I/O related tests (you didn't explain with details what you did with the LVM volumes), is there any chance that the behavior you noticed is related to the natural OS page cache?

I'm thinking about the Linux standard behavior of using (sometimes almost all) available memory (that is, memory that is not reserved by applications) to cache disk regions that were recently accessed, in order to speed up IOs.

Of course, active applications have priority over this cache, so in case a process needs more ram, the cache is dynamically shrinked.

There is a lot of documentation about Linux page cache on the web, here is one.

1
  • That was my first thought as well, but no, I am certain it is not. As you said, cache is automatically shinked when memory is needed. In this case, the server effectively runs out of memory. Besides, page cache shows as orange in htop. Thanks for your answer though.
    – SharkWipf
    Jul 3, 2013 at 13:57
2

I have the same problem, but I checked the kernel parameters located under /proc/sys/vm/ : overcommit_memory = Default 0 overcommit_ratio = Default 50 max_map_count = Default 65530

And in my case the parameter, root of my problems nr_hugepages, i haved the custom value, but change it for default

nr_hugepages = 0

For more information see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-memory-captun

Sorry for my english :(

1
  • You are a lifesaver Jeant! Hugepages were an issue also for me.
    – charlie_pl
    May 18, 2020 at 9:23
1

For me it seemed to be the page cache not clearing itself.

sync; echo 1 | sudo tee /proc/sys/vm/drop_caches

Gave me 30gB back after a few seconds.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .