Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have some 64 bits Linux boxes (RHEL6) running on Microsoft Windows 2012 Server (Hypervisor) with the same problem. Those Linux guest servers are running in MS Windows Cloud (Hyper-V) with 16 hosts servers with 256 GB of RAM each one.

They start with the following memory usage:

# free -m
             total       used       free     shared    buffers     cached
Mem:         48259        653      47606          0         19        106
-/+ buffers/cache:        527      47732
Swap:        13999          0      13999



# cat /proc/meminfo
MemTotal:       49418204 kB
MemFree:        48749868 kB
Buffers:           20080 kB
Cached:           108564 kB
SwapCached:            0 kB
Active:           149652 kB
Inactive:          98856 kB
Active(anon):     120124 kB
Inactive(anon):     1884 kB
Active(file):      29528 kB
Inactive(file):    96972 kB
Unevictable:           4 kB
Mlocked:               4 kB
SwapTotal:      14335992 kB
SwapFree:       14335992 kB
Dirty:               788 kB
Writeback:             0 kB
AnonPages:        122196 kB
Mapped:            39844 kB
Shmem:              2132 kB
Slab:              51832 kB
SReclaimable:      14696 kB
SUnreclaim:        37136 kB
KernelStack:        5656 kB
PageTables:        15840 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    39045092 kB
Committed_AS:     490856 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      131964 kB
VmallocChunk:   34359602252 kB
HardwareCorrupted:     0 kB
AnonHugePages:     28672 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        8128 kB
DirectMap2M:    50323456 kB

(Ordered by RES)

# top
top - 11:26:52 up 1 min,  1 user,  load average: 1.71, 0.73, 0.27
Tasks: 609 total,   1 running, 608 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.1%hi,  0.0%si,  0.0%st
Mem:  49418204k total,   674472k used, 48743732k free,    20472k buffers
Swap: 14335992k total,        0k used, 14335992k free,   111720k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1859 root      20   0  103m  28m 5784 S  0.0  0.1   0:00.85 Xvnc
 2037 root      20   0  508m  25m 8876 S  0.7  0.1   0:01.18 gnome-settings-
 2064 root      20   0  373m  17m  12m S  0.0  0.0   0:01.11 nautilus
 2141 root      20   0  464m  16m  12m S  0.0  0.0   0:00.14 clock-applet
 2063 root      20   0  319m  14m  10m S  0.0  0.0   0:00.28 gnome-panel
 2082 root      20   0  307m  12m 9100 S  0.0  0.0   0:00.11 nm-applet
 2139 root      20   0  381m  12m 9748 S  0.0  0.0   0:00.08 gdm-user-switch
 2093 root      20   0  442m  11m 9104 S  0.0  0.0   0:00.13 gnome-volume-co
 2116 root      20   0  299m  11m 9476 S  0.0  0.0   0:00.10 wnck-applet
 2118 root      20   0  307m  11m 8768 S  0.0  0.0   0:00.06 trashapplet
...

And 6 minutes later the memory is consumed by an unknown process or by the kernel:

uptime ; free -m
 11:31:52 up 6 min,  1 user,  load average: 1.05, 0.93, 0.47
             total       used       free     shared    buffers     cached
Mem:         48259      25296      22963          0         21        160
-/+ buffers/cache:      25115      23144
Swap:        13999          0      13999

]# cat /proc/meminfo
MemTotal:       49418204 kB
MemFree:        23514240 kB
Buffers:           21600 kB
Cached:           164428 kB
SwapCached:            0 kB
Active:           210768 kB
Inactive:         108108 kB
Active(anon):     133036 kB
Inactive(anon):     2332 kB
Active(file):      77732 kB
Inactive(file):   105776 kB
Unevictable:           8 kB
Mlocked:               8 kB
SwapTotal:      14335992 kB
SwapFree:       14335992 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        132892 kB
Mapped:            41976 kB
Shmem:              2516 kB
Slab:              52624 kB
SReclaimable:      17628 kB
SUnreclaim:        34996 kB
KernelStack:        5752 kB
PageTables:        15756 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    39045092 kB
Committed_AS:     654848 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      131964 kB
VmallocChunk:   34359602252 kB
HardwareCorrupted:     0 kB
AnonHugePages:     45056 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        8128 kB
DirectMap2M:    50323456 kB

(Ordered by RES)

# top
top - 11:32:45 up 7 min,  1 user,  load average: 1.02, 0.94, 0.50
Tasks: 607 total,   1 running, 606 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.1%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  49418204k total, 25904096k used, 23514108k free,    21600k buffers
Swap: 14335992k total,        0k used, 14335992k free,   164428k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1859 root      20   0  103m  28m 6176 S  0.0  0.1   0:00.87 Xvnc
 2037 root      20   0  508m  25m 8916 S  0.0  0.1   0:01.19 gnome-settings-
 2064 root      20   0  373m  17m  12m S  0.0  0.0   0:01.61 nautilus
 2141 root      20   0  464m  16m  12m S  0.0  0.0   0:00.15 clock-applet
 2063 root      20   0  319m  14m  10m S  0.0  0.0   0:00.29 gnome-panel
 1788 root      20   0  779m  13m 5808 S  0.0  0.0   0:00.18 scxcimserver
 2082 root      20   0  307m  12m 9100 S  0.0  0.0   0:00.11 nm-applet
 2139 root      20   0  381m  12m 9748 S  0.0  0.0   0:00.08 gdm-user-switch
 2093 root      20   0  442m  11m 9104 S  0.0  0.0   0:00.13 gnome-volume-co
 2116 root      20   0  299m  11m 9476 S  0.0  0.0   0:00.10 wnck-applet
...

The sum of RSS memory reported by ps in MB is about 388:

# ps aux | awk '{sum+=$6} END {print sum / 1024}'
387.898

What else can I check to find out what is consuming the memory?

share|improve this question
    
Is your hypervisor over-provisioned? –  Nathan C Jan 30 at 19:14
1  
Now that I see it better, 49418204 is ~48 GB of memory...are you allocating your entire host's RAM to this guest or something? –  Nathan C Jan 30 at 19:22
    
No, those guests are running in a MS Windows farm (Cloud) with 16 servers as Host. Each of those servers has 256 GB of RAM. –  jorgernan Jan 30 at 20:56

1 Answer 1

up vote 7 down vote accepted

It sounds like Hyper-V is stealing the memory because your host is over-provisioned, a process called memory ballooning (or Dynamic Memory in the Hyper-V world).

share|improve this answer
    
This is the most likely scenario. Dynamic memory will cause the exact behavior described in an overprovisioned host, or in an underutilized guest. The kernel module that enables it on Linux is hv_balloon and you can check for it with lsmod | grep hv_ –  MDMarra Jan 30 at 19:28
    
Yes, it uses those kernel modules: # lsmod | grep hv_ hv_balloon 12019 0 [permanent] hv_netvsc 23702 0 hv_utils 9149 0 hv_storvsc 11323 2 hv_vmbus 144850 6 hv_balloon,hid_hyperv,hv_netvsc,hv_utils,hyperv_fb,hv_storvsc –  jorgernan Jan 31 at 13:20
    
Is that the expected behavior? Consuming memory on guest level or the guest VM should be affected by a reduced MemTotal? –  jorgernan Jan 31 at 13:23
    
@jorgernan Ballooning works by "stealing" guest RAM for the host. The kernel reports memory used even though it's not really "used" by a process. This is probably because your guest is underutilized so it allocates the RAM someplace else. –  Nathan C Jan 31 at 13:30
    
@jorgernan if you don't want that to happen, either disable dynamic memory or raise the minimum value. That said, if your guest needs the memory and the host has available resources, your guest will be able to use the RAM. All that it means at this point is you've drastically over-provisioned your guest. –  MDMarra Feb 1 at 16:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.