6

I have a desktop running Ubuntu Quantal using OpenStack Folsom on an Intel i5 with 32 GB RAM and 2 GB swap. I'm running 7 VMs each sized like a EC2 m1.small, so 1.7 GB RAM each. I'm using KVM.

As I get up to running 5 or 6 concurrently, the host starts to swap them out:

top - 23:45:42 up 3 days,  1:51, 10 users,  load average: 0.37, 0.75, 1.15
Tasks: 418 total,   2 running, 413 sleeping,   3 stopped,   0 zombie
%Cpu(s):  8.8 us,  2.1 sy,  0.0 ni, 88.8 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem:  32864580 total, 32586956 used,   277624 free,   574236 buffers
KiB Swap:  1998844 total,  1113352 used,   885492 free, 16498252 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND           
24652 libvirt-  20   0 4169m 1.7g 7756 S   3.6  5.4   4:49.37 kvm               
25233 libvirt-  20   0 4450m 1.6g 7756 S   1.2  5.2   4:35.12 kvm               
25589 libvirt-  20   0 4163m 1.6g 7756 S   2.4  5.1   4:40.31 kvm               
 6562 root      39  19 2935m 658m 7460 S   0.0  2.1 100:05.62 java              
28393 libvirt-  20   0 4149m 624m 7756 S   0.0  1.9   2:25.01 kvm               
28106 libvirt-  20   0 4170m 617m 7756 S   0.0  1.9   2:18.17 kvm               
26519 libvirt-  20   0 4167m 590m 7756 S   0.0  1.8   2:22.16 kvm               
29399 libvirt-  20   0 4159m 589m 7756 S   0.0  1.8   2:19.94 kvm               


$ free -m
             total       used       free     shared    buffers     cached
Mem:         32094      31868        225          0        559      16175
-/+ buffers/cache:      15134      16959
Swap:         1951       1087        864


# /tmp/swap-used.sh |grep kvm
PID=944 - Swap used: 0 - (kvm-irqfd-clean )
PID=24652 - Swap used: 102468 - (kvm )
PID=25233 - Swap used: 108644 - (kvm )
PID=25589 - Swap used: 155768 - (kvm )
PID=26519 - Swap used: 192216 - (kvm )
PID=28106 - Swap used: 150796 - (kvm )
PID=28393 - Swap used: 208488 - (kvm )
PID=29399 - Swap used: 187388 - (kvm )

I've already tried setting swapiness to 20, then 10 and finally to 0, none of which has made a difference:

# cat /proc/sys/vm/swappiness 
0

I haven't rebooted the host since changing it from 60 to 0 (do I need to reboot)? I've also turned off swap completely with /sbin/swapoff -a; /bin/swapon -a. Immediately after re-enabling swap, I see this:

$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  0  11384 247572 652736 16539748   11   10   291   228   13   12  7  3 87  3
 0  0  12968 234360 652756 16554432    0  317  8576  1240 3508 5470 17  2 75  5
 1  0  17068 243512 652768 16559508    0  820  9448  1216 3687 4845 20  2 77  2
 1  0  20040 233300 652772 16576152    0  594 12262   677 4436 5063 29  2 68  1
 1  0  22764 219156 652792 16594448    6  546 11962   727 3870 4559 28  1 68  2
 3  0  40832 229384 652776 16602440    0 3614 58404  4176 2051 6231 21  2 66 10
 1  0  52420 232236 652784 16613320    0 2318 42174  2512 1819 4026 15  2 77  6

I've got 15 GB of free memory that can be used without having the processes swapped out.

7
  • If you merely set vm.swappiness, that doesn't automatically swap in anything already in swap. Try swapoff -a; swapon -a to do that. Feb 12, 2013 at 8:04
  • I've done that a couple of times and it goes back to swapping the VMs immediately. Feb 12, 2013 at 8:17
  • 2
    Is it only ever swapping out (so) and never/rarely in (si)? If so, it's nothing to worry about. it's chunks of memory that are just not being used and thus the OS decided that the page cache is more important. That said, KSM might be also extremely useful for you. It should lower the memory pressure if each guest OS is similar.
    – R. S.
    Feb 12, 2013 at 8:35
  • 4
    Because your OS is far better at memory management than you believe you are. Feb 12, 2013 at 10:04
  • @TomO'Connor I've done what the linked to article suggests on changing swappiness and it had no effect, so this isn't a dup. Feb 12, 2013 at 18:31

1 Answer 1

8

I've got 15 GB of free memory that can be used without having the processes swapped out.

No, you don't.

           total       used       free     shared    buffers     cached
Mem:       32094      31868        225          0        559      16175

You have 225MB of memory free.

KiB Swap: 1998844 total, 1113352 used, 885492 free, 16498252 cached

Look at all that wonderful cache the system has made, in part by getting unused junk out of precious RAM. Why are you trying to make it waste precious RAM?

If you have 1GB of data in RAM that hasn't been used in hours, which makes more sense:

  1. You swap that out to disk when you're not busy and have 1GB more cache to use.

  2. You keep it in memory. You have 1GB less cache to use, and if that memory is ever needed for something else, you'll have to swap it out while you are busy.

But if you think you know more about memory management than the people who wrote your operating system's memory management logic, go ahead and keep trying to find that magic "do everything faster and better" switch.

14
  • 3
    (It's a useful observation, but don't you think the tone could be more constructive?) Feb 12, 2013 at 9:23
  • 11
    I've tried every tone I know of for several years now. I've yet to find one that works. People continue to insist on thwanging delicately tuned knobs in various directions in response to imagined problems no matter what. But I'm not bitter. Feb 12, 2013 at 9:35
  • 2
    This answer is incorrect in regards to the amount of "free" memory that the OS can reallocate to different tasks, see linuxatemyram.com . The OS will use all available memory and over time the top output will always show free memory close to 0 which is what you want. But if you look at the free -m output on the -/+ buffers/cache line above it shows that if you account for the buffer cache where pages can be dropped, there's 16959 MB of memory that the OS can use. I would rather the OS leave the VMs in memory and drop buffer cache pages than otherwise. Feb 12, 2013 at 18:17
  • 1
    That cache is being used to make the active VMs faster. What you are proposing is that you make all your VMs equally slow - instead of allowing the OS to make the active VMs faster and the innactive VMs slower. This doesn't make sense unless you're trying to provide some sort of incredibly tight SLA.
    – Chris S
    Feb 12, 2013 at 18:45
  • 1
    @ChrisS I have 7 VMs at 1.7 GB each = 11.9 GB on a host system with 32 GB of RAM, so I can afford to keep them all in RAM. Feb 12, 2013 at 19:11

Not the answer you're looking for? Browse other questions tagged .