3

We've been having some Linux memory usage problems on some of our servers. free, top, ps, and vmstat are all showing 261M free out of 1G (~25%). However, adding up the amount of memory shown per process in top shows that we should be using very little memory. The cache has only about 48M and 0K in the buffer so that makes the total memory available 309M, still considerably less than we should have. I have included a screen shot of our top output at http://www.workxpress.com/sites/default/files/top.png (sorted by memory). This is my first post on serverfault so I couldn't include the image in my post :-P

After trying several Google searches, I am no where further than I was. The majority of the results told me the same thing, the memory is being used for cache and buffers (which I already knew). But top and free clearly don't show that. Any help is greatly appreciated.

Thanks,
JamesArmes

Update:
I am including our /proc/meminfo from a more recent occurrence of this issue: 

MemTotal:      1028636 kB
MemFree:         30056 kB
Buffers:             0 kB
Cached:          28732 kB
SwapCached:     154684 kB
Active:         265328 kB
Inactive:        19416 kB
SwapTotal:     2097144 kB
SwapFree:      1758196 kB
Dirty:              36 kB
Writeback:           0 kB
AnonPages:      240260 kB
Mapped:          11996 kB
Slab:            23008 kB
SReclaimable:    11976 kB
SUnreclaim:      11032 kB
PageTables:       5636 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   2611460 kB
Committed_AS:  1099080 kB
VmallocTotal: 34359738367 kB
VmallocUsed:      7204 kB
VmallocChunk: 34359731091 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     2048 kB

Below is our /etc/sysctl.conf (with all of the commented lines removed):

kernel.printk = 4 4 1 7                            
kernel.maps_protect = 1                                            
fs.inotify.max_user_watches = 524288
vm.mmap_min_addr = 65536
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1

I hope this helps.

Improve this question
1
  • 1
    Just so you know, I had exactly this problem with a VMware server, three months ago. We fixed it by turning off the memory limit. I suggest you read up about how VMware handles memory. What is happening is expected. If you get your ESX admin to look at "Memory Balloon Average" in the performance/memory tab of the VCI, it will show where your "missing" memory is.
    – Swoogan
    Oct 29, 2009 at 19:33

4 Answers 4

Reset to default
3

Since I see vmware-guestd running, I'll assume that this is a VM. Assuming ESX, go into the VIC and right-click the machine. Click Edit Settings... and select the Resources tab. Click Memory and ensure that Unlimited is check under Limit.

What is likely happening is that you have a limit on the memory, and when the machine uses lots of memory (approaches or exceeds the limit), the vmware guest tools keep the memory "used" rather than releasing it back to ESX, in case it's needed again.

UPDATE:

"The server allows you to power on a virtual machine only if the CPU and memory reservation is available... When resources are not used, the ESX Server host makes them available to other virtual machines."

http://www.vmware.com/pdf/vi3_esx_resource_mgmt.pdf

It does this by way of the "balloon" driver (vmmemctl) which is part of the guest tools. Say your machine is using 100MB of ram, then you run some program and it jumps up to 500MB used. Now you stop that program and expect the RAM to go back down to 100MB. However, it doesn't. This is because, in order for ESXi to reclaim the memory back from the guest OS (which it does even when there is a reservation) it must use the balloon driver. To make the OS "act" as though it has less memory, the balloon driver "uses" the memory that ESXi wants, so that the OS can't use it.

IOW, even with a reservation the guest is only using as much host RAM as it is using. When it uses more, ESX allocates more host RAM which the guest now thinks it always has available. To "convince" the guest that the new ram is gone again, the balloon driver uses it up. The reservation simply means "Don't start the VM unless the physical machine has 1GB free to load the guest into" and not "Give the guest 1GB whether it uses it or not."

Improve this answer
2
  • You are correct in your assumption that we are using ESX (ESXi to be exact). We don't have access to the VIC for this machine, but we know that the limit for this machine is 1G with a reserve of 1G. There is a second machine that is limited to and reserves the majority of what's left on the machine.
    – JamesArmes
    Oct 29, 2009 at 15:21
  • We took off the limit and it made no difference. However, we did find that the balloon was, on average, taking roughly the same amount of memory as we were missing. Thanks to the document you linked to, we found that we can shut off the ballooning for a specific machine by setting sched.mem.maxmemctl to 0. By doing so, we were able to resolve this issue. While I wouldn't normally recommend this, the ESX server this machine is on is dedicated to two VMs.
    – JamesArmes
    Oct 31, 2009 at 16:33
1

Try:

cat /proc/meminfo

It gives more details of the memory usage.

Improve this answer
1
  • We tried this, but it didn't lead us anywhere.
    – JamesArmes
    Oct 29, 2009 at 16:00
1

Try to debug memory usage with the script ps_mem.py

Improve this answer
0

You may have kernel memory reservation. Specifically, check hugepages parameters in /proc/meminfo.

And please post your /etc/sysctl.conf if this suggestion doesn't help.

Improve this answer
4
  • our Hugepagesize is listed as 2048K. I have updated my post to include both /proc/meminfo and /etc/sysctl.conf.
    – JamesArmes
    Oct 29, 2009 at 17:38
  • Wild guess: reduce max_user_watches.
    – Max Alginin
    Oct 29, 2009 at 17:59
  • @ynguldyn We don't use inotify so I don't have any idea what a reasonable value for this would be. Do you have any suggestions?
    – JamesArmes
    Oct 29, 2009 at 18:31
  • Remove the line completely then.
    – Max Alginin
    Oct 29, 2009 at 19:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .