1

This question already has an answer here:

I found a bizare think on my server. The swap used is significantly higher than the total swap.

free -m
             total       used       free     shared    buffers     cached
Mem:          1869       1788         80         71         35        487
-/+ buffers/cache:       1265        603
Swap:         2046 1759218603       9146

The swapon -s shows that the swap size is 2.0 GB:

 swapon -s
Filename                                Type            Size    Used    Priority
/dev/sdb1                               partition       2096124 46584   -1

Atop swap usage

For info the top swap consumers are the folowing:

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r |head -n 10
scanner 11988 kB
scanner 11944 kB
scanner 11936 kB
scanner 11932 kB
php-fpm 4428 kB
log 2920 kB
snmpd 2700 kB
cma 1996 kB
cma 1996 kB
cma 1996 kB

Meminfo:

cat /proc/meminfo
MemTotal:        1914596 kB
MemFree:          672796 kB
Buffers:           19100 kB
Cached:           387924 kB
SwapCached:         5348 kB
Active:           470276 kB
Inactive:         446516 kB
Active(anon):     310168 kB
Inactive(anon):   278548 kB
Active(file):     160108 kB
Inactive(file):   167968 kB
Unevictable:        3896 kB
Mlocked:            3896 kB
SwapTotal:       2096124 kB
SwapFree:        9384400 kB
Dirty:                48 kB
Writeback:             0 kB
AnonPages:        510888 kB
Mapped:            44264 kB
Shmem:             76232 kB
Slab:             261036 kB
SReclaimable:     217824 kB
SUnreclaim:        43212 kB
KernelStack:        4272 kB
PageTables:        14600 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3053420 kB
Committed_AS:    1391372 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      281060 kB
VmallocChunk:   34359452940 kB
HardwareCorrupted:     0 kB
AnonHugePages:    172032 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        8192 kB
DirectMap2M:     2088960 kB

[EDIT] As asked here is the free without the -m argument and the uname -a

free
             total       used       free     shared    buffers     cached
Mem:       1914596    1741152     173444      75800       8740     377320
-/+ buffers/cache:    1355092     559504
Swap:      2096124 1801439850   10696044

uname -a
Linux myserver 2.6.32-573.3.1.el6.x86_64 #1 SMP Thu Aug 13 22:55:16 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

For info i also found this in my dmesg:

Out of memory: Kill process 28579 (php-fpm) score 137 or sacrifice child
Killed process 28579, UID 516, (php-fpm) total-vm:1439832kB, anon-rss:68204kB, file-rss:4072kB
hpet1: lost 1 rtc interrupts
vmtoolsd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
vmtoolsd cpuset=/ mems_allowed=0
Pid: 2024, comm: vmtoolsd Tainted: G        W  -- ------------    2.6.32-573.3.1.el6.x86_64 #1
Call Trace:
 [<ffffffff810d6dd1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
 [<ffffffff8112a5d0>] ? dump_header+0x90/0x1b0
 [<ffffffff81232cbc>] ? security_real_capable_noaudit+0x3c/0x70
 [<ffffffff8112aa52>] ? oom_kill_process+0x82/0x2a0
 [<ffffffff8112a991>] ? select_bad_process+0xe1/0x120
 [<ffffffff8112ae90>] ? out_of_memory+0x220/0x3c0
 [<ffffffff8113786c>] ? __alloc_pages_nodemask+0x93c/0x950
 [<ffffffff810f5709>] ? delayacct_end+0x89/0xa0
 [<ffffffff811275f0>] ? sync_page_killable+0x0/0x40
 [<ffffffff8117035a>] ? alloc_pages_vma+0x9a/0x150
 [<ffffffff81163882>] ? read_swap_cache_async+0xf2/0x160
 [<ffffffff811643d9>] ? valid_swaphandles+0x69/0x160
 [<ffffffff81163977>] ? swapin_readahead+0x87/0xc0
 [<ffffffff81152abd>] ? handle_pte_fault+0x6dd/0xb20
 [<ffffffff811b45c0>] ? mntput_no_expire+0x30/0x110
 [<ffffffff81153199>] ? handle_mm_fault+0x299/0x3d0
 [<ffffffff8119fd7b>] ? putname+0x2b/0x40
 [<ffffffff81155bb6>] ? find_vma+0x46/0x80
 [<ffffffff8104f156>] ? __do_page_fault+0x146/0x500
 [<ffffffff81290271>] ? cpumask_any_but+0x31/0x50
 [<ffffffff81197750>] ? vfs_fstatat+0x50/0xa0
 [<ffffffff81014a19>] ? read_tsc+0x9/0x10
 [<ffffffff8106343d>] ? thread_group_times+0x3d/0x120
 [<ffffffff811978cb>] ? vfs_stat+0x1b/0x20
 [<ffffffff8153e95e>] ? do_page_fault+0x3e/0xa0
 [<ffffffff8153bd05>] ? page_fault+0x25/0x30
Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 110
CPU    1: hi:  186, btch:  31 usd:  62
CPU    2: hi:  186, btch:  31 usd:  48
CPU    3: hi:  186, btch:  31 usd:  78
active_anon:322906 inactive_anon:108241 isolated_anon:480
 active_file:1427 inactive_file:1298 isolated_file:0
 unevictable:994 dirty:0 writeback:1 unstable:0
 free:13249 slab_reclaimable:2892 slab_unreclaimable:9700
 mapped:3333 shmem:10113 pagetables:6259 bounce:0
Node 0 DMA free:8352kB min:332kB low:412kB high:496kB active_anon:3544kB inactive_anon:3652kB active_file:0kB inactive_file:8kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15300kB mlocked:0kB dirty:0kB writeback:0kB mapped:8kB shmem:56kB slab_reclaimable:16kB slab_unreclaimable:36kB kernel_stack:0kB pagetables:52kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11333 all_unreclaimable? yes
lowmem_reserve[]: 0 2004 2004 2004
Node 0 DMA32 free:44644kB min:44720kB low:55900kB high:67080kB active_anon:1288080kB inactive_anon:429184kB active_file:5708kB inactive_file:5184kB unevictable:3976kB isolated(anon):2048kB isolated(file):0kB present:2052192kB mlocked:3976kB dirty:0kB writeback:4kB mapped:13324kB shmem:40396kB slab_reclaimable:11552kB slab_unreclaimable:38764kB kernel_stack:4480kB pagetables:24984kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2629632 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 12*4kB 14*8kB 6*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8352kB
Node 0 DMA32: 435*4kB 377*8kB 269*16kB 344*32kB 142*64kB 39*128kB 23*256kB 3*512kB 1*1024kB 1*2048kB 0*4096kB = 44644kB
13751 total pagecache pages
196 pages in swap cache
Swap cache stats: add 35252293, delete 35252097, find 25799626/28229091
Free swap  = 8734852kB
Total swap = 2096124kB
524272 pages RAM
45687 pages reserved
34271 pages shared
456140 pages non-shared

Whis is it so? Is there something anormal on the server?

marked as duplicate by Michael Hampton Oct 28 '15 at 12:53

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • You should include the contents of /proc/swaps and /proc/meminfo in the question. – kasperd Oct 20 '15 at 6:23
  • @kasperd Its done. for info the /proc/swaps will get you the same info as swapon -s – Kheshav Sewnundun Oct 20 '15 at 6:44
  • This is very curious and I think it's worth trying to figure out. However, when that's all done, how about trying to recreate the swap partition? – Ryan Babchishin Oct 20 '15 at 7:57
  • Exactly what sort of 'server' is this? – Michael Hampton Oct 20 '15 at 14:13
  • Its a webserver with apache and php-fpm. – Kheshav Sewnundun Oct 20 '15 at 14:18
3

The solution was to Upgrade to kernel-2.6.32-573.7.1.el6 or higher.

As mentioned in Kernel Bugfix update and Solution Article a yum update did the work

Below is an extract of the bug report of kernel-2.6.32-573.1.1.el6.x86_64 that caused the Swap free to be greater than the swap total:

A previous change in the get_swap_page() locking removed the use of the swap_lock spinlock. This could cause nr_swap_pages corruption and invalid SwapFree information in the /proc/meminfo file, where the size of SwapFree could exceed the size of SwapTotal. This update uses an atomic variable for nr_swap_pages, and the size of SwapFree in /proc/meminfo is now correct. (BZ#1259362)

2

Looks like this problem has already been mentioned on ServerFault at "free reports bogus amount of swap used" and is also described by Red Hat in their (non-public) errata tracker at https://access.redhat.com/solutions/1571043. I don't have access to the full solution text from Red Hat, so can't give you any more details.

  • Directed me to the right track Thank you. The solution was to upgrade to kernel-2.6.32-573.7.1.el6 – Kheshav Sewnundun Oct 28 '15 at 12:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.