1

This server is swapping using 100% (15G swap disk) while it still has 30G free, I've been looking arround but can't figure why this is happening, my understanding was not to use swap if memory is still available.

# cat /proc/meminfo 
MemTotal:     131937984 kB
MemFree:      29649644 kB
Buffers:        347424 kB
Cached:       55013896 kB
SwapCached:    2180808 kB
Active:       42065900 kB
Inactive:     15782332 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     131937984 kB
LowFree:      29649644 kB
SwapTotal:    16779884 kB
SwapFree:      2165300 kB
Dirty:            1908 kB
Writeback:           0 kB
AnonPages:      323104 kB
Mapped:       31100628 kB
Slab:           408604 kB
PageTables:    1737344 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  61777356 kB
Committed_AS: 52238240 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    382524 kB
VmallocChunk: 34359355491 kB
HugePages_Total: 20480
HugePages_Free:  20480
HugePages_Rsvd:      0
Hugepagesize:     2048 kB


# top
top - 14:33:16 up 27 days, 21:25,  7 users,  load average: 3.47, 4.39, 4.34
Tasks: 669 total,   2 running, 661 sleeping,   0 stopped,   6 zombie
Cpu(s):  7.6%us,  1.5%sy,  0.0%ni, 90.2%id,  0.6%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  131937984k total, 101503048k used, 30434936k free,   347544k buffers
Swap: 16779884k total, 14608832k used,  2171052k free, 54216540k cached

#  vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0 14608504 30438520 347560 54216984    1    1   503   262    0    0  8  2 90  1  0
 3  0 14608504 30439016 347560 54217024    0    0    88   403 1434 4884  0  0 99  0  0
 2  0 14608468 30439760 347560 54217060    0    0    10   588 2381 5297  7  1 93  0  0
 0  0 14608468 30439884 347560 54217060    0    0     0    76 1429 4768  5  0 94  0  0
 1  0 14608448 30440180 347560 54217048   32    0    66   230 3371 4872  4  1 95  0  0
 6  0 14608420 30440668 347560 54217076   32    0    42   320 2439 4860  6  1 93  0  0
 3  0 14608412 30441384 347560 54217116    0    0    58   520 2128 4899  6  1 93  0  0
 4  0 14608412 30441504 347560 54217116    0    0     0    58 1355 4477 11  1 88  0  0
 3  0 14608392 30441844 347560 54217012  128    0   158    16 1491 4374 13  1 86  0  0
 5  1 14608352 30441512 347560 54216924  160    0   296   640 2748 5279 15  2 83  0  0
 4  0 14608324 30442132 347564 54217112   32    0    90   502 2493 4878 13  1 86  0  0
 8  0 14608296 30437288 347568 54217204    0    0     8   724 2243 5185 12  1 86  0  0

# cat /proc/sys/vm/swappiness
60

bash-3.2# grep ^[a-z]  /etc/sysctl.conf 
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.default.log_martians = 0
kernel.panic_on_oops = 1
kernel.panic = 5
fs.aio-max-nr = 399360 
kernel.exec-shield = 0
kernel.randomize_va_space = 0
vm.nr_hugepages = 20480
net.ipv4.tcp_keepalive_time     300

An option could be change swappiness from 60 to a lower number

4 Answers 4

1

I have a Sybase server with CentOS 5 which causes no swap. The trick is to set swappiness to 0 and to left enough memory for OS. The server has 16GB (10GB for Sybase and 6GB for OS and other services). Start with a smaller Sybase cache and increase it progressively.

[root@db2 ~]# tail -2 /etc/sysctl.conf 
# Set Swappiness
vm.swappiness = 0
0

You're probably running software that maintains big caches in process memory. Such pages will be swapped out when another memory-hungry task is run. The memory will not be swapped back until necessary, so when the other memory-hungry task is done, there will be a lot of free memory that won't be filled until necessary.

5
  • The box is running sybase + VCS, but Sybase was started with 32G memory cap. Is there a way to force the linux not to use the swap if not really needed? Also there is still 29G of mem not in use, I would be happy to track it down, not sure what to do next.
    – user420329
    Jun 16, 2011 at 13:52
  • Lost of memory seems to go into caching, is that a normal behavior?
    – user420329
    Jun 16, 2011 at 13:54
  • Seems like you describe normal linux caching behaviour :) Are you sure there has not been a memory intensive task in the past? Jun 16, 2011 at 14:04
  • A database system may be slower when it expects quick access to derived information in a cache. Re-deriving it might be faster. Dropping the caches should cause them to be regenerated in RAM. Jun 16, 2011 at 14:10
  • sar is showing lots of pagin pageout, I need to identify which process is doing it (likely to be sybase) but really wonder why the system is not using the 30G of free memory and goes for the 15G of swap.
    – user420329
    Jun 16, 2011 at 14:50
0

A friend had this exact problem. I'm assuming you have a multicore system. Being that the case, Linux divides the total physical RAM per processor (ie. if you have a 4core and 16 gigs of ram, it reserves 4gigs per core). When all available RAM is used up for a particular processor, it proceeds to use swap space, instead of taking free RAM from the other processors.

Let me contact him to see what is the exact solution he got, maybe this answer can point to the correct direction in the meanwhile.

2
  • That would make lot of sense! This server has 32 core and 128G of ram. An other solution would have been to use huge_pages but it's not working with this version of Sybase, that would be great if you frirend has the solution.
    – user420329
    Jun 16, 2011 at 16:21
  • -1 That answer is wrong on many levels.
    – kubanczyk
    Feb 24, 2018 at 0:55
0

If you have a single process which has allocated a huge amount of memory on a NUMA system, it may be that it's only letting it have memory belonging to one NUMA node or something.

Have a read of this:

http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/

The bottom line is: if you can numactl it to use all the memory, it may be better.

However, telling your database to use 32G of 32G seems optimistic, as there will be significant overhead. In particular, on x86_64 the page tables will take up a lot of space (maybe .5 G?) and other parts of the system will need some space. I'd think that about 75% was a better setting.

You must log in to answer this question.