12

I have virtual linux (Fedora 17) server with 28GB RAM and 2GB swap. The server is running a MySQL DB that is set up to use most of the RAM.

After some time running the server starts to use swap to swap out unsued pages. That is fine as my swappiness is at default 60 and it is the expected behaviour.

The strange thing is that the number in top/meminfo does not correspond with info from processes. I.e. the server is reporting these numbers:

/proc/meminfo:
SwapCached:        24588 kB
SwapTotal:       2097148 kB
SwapFree:         865912 kB

top:
Mem:  28189800k total, 27583776k used,   606024k free,   163452k buffers
Swap:  2097148k total,  1231512k used,   865636k free,  6554356k cached

If I use the script from https://serverfault.com/a/423603/98204 it reports reasonable numbers (few MBs swapped by bash'es, systemd, etc) and one big allocation from MySQL (I omitted a lot of output lines):

892        [2442] qmgr -l -t fifo -u
896        [2412] /usr/libexec/postfix/master
904        [28382] mysql -u root
976        [27559] -bash
984        [27637] -bash
992        [27931] SCREEN
1000       [27932] /bin/bash
1192       [27558] sshd: admin@pts/0
1196       [27556] sshd: admin [priv]
1244       [1] /usr/lib/systemd/systemd
9444       [26626] /usr/bin/perl /bin/innotop
413852     [31039] /usr/libexec/mysqld --basedir=/usr --datadir=/data/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/data/mysql/err --open-files-limit=8192 --pid-file=/data/mysql/pid --socket=/data/mysql/mysql.sock --port=3306
449264   Total Swap Used

So if I get the script output right the total swap usage should be 449264K = ca. 440MB with mysql using ca. 90% of the swap.

The question is why this differs so much from the top and meminfo numbers? Is there any way how to "dump" swap info to see what is actually in it instead of summing the swap usages from all processes?

When analyzing the issue I came up with different ideas but they all seems to be wrong:

  1. The script output is not in KB. Even if it would be in 512 or 4KB units it won't match. Actually the ratio (1200:440) is about 3:1 which is "strange" number.
  2. There are some pages in swap that are somehow shared between processes as mentioned in https://serverfault.com/a/477664/98204 . If this is true how can I find the actual number of memory used like this? I mean it would need to make cca 800MB difference. And that does not sound right in this scenario.
  3. There are some "old" pages in swap used by processes that already finished. I would not mind that if I were able to find out how much is this "freeable" swap.
  4. There are pages in swap that has been swapped back to memory and are in swap just in case they did not change in RAM and need to be swapped out again as mentioned in https://serverfault.com/a/100636/98204 . But the SwapCached value is only 24MB.

The strange thing is that the swap usage is slowly increasing while the sum output from the script is roughly the same. In last 3 days the swap used increased from 1100MB to current 1230MB while the sum increased from 430MB to current 449MB (ca.).

The server has enough free(able) RAM so I could just turn off the swap and turn it back on. Or I could probably set swappiness to 0 so the swap would get used only if the is no other way. But I would like to solve the issue or at least find out what is the cause of this.

5
  • Like you say you should just set vm.swappiness = 0 (or 1) and swapoff && swapon
    – HTTP500
    Nov 5, 2013 at 15:40
  • But that would not solvethe issue. I am assuming that the swap would start increasing again if I set swappines back to 60 or would not be used at all if I keep it at 0 or 1 (unless the server goes out of memory) Nov 5, 2013 at 15:53
  • If it's a DB Server it should only use swap in an emergency so you should always set it to 0 (or 1).
    – HTTP500
    Nov 5, 2013 at 19:12
  • Thats true and thats probably what I am gonna do if I do not find the cause of this issue... On the other hand there is a lot of small DBs on the server that are used very sporadically and I liked the idea of them being swapped out by system when they are not in use... However I suppose that MySQL will be able to handle it on its own... Nov 5, 2013 at 19:52
  • Mysql does a pretty good job of managing its own caches, but that's based on assumptions about what is in fact in memory and what's not. If you try to double-guess it by using swap memory, you're just impairing mysql's ability to decide what needs to be cached and what doesn't. Turn swap off. IF you hit swap, that's a tuning failure. Adjust your cache sizes so swap never happens, but short of that, you want to utilise all available physical memory.
    – mc0e
    Nov 6, 2013 at 14:17

2 Answers 2

9

Fedora 18 and up have smem in the repos. You could download the python script and install from source.

Here's a sample output (somewhat snipped and anonymized) from my machine:

# smem -s swap -t -k -n
  PID User     Command                         Swap      USS      PSS      RSS 
20917 1001     bash                               0     1.1M     1.1M     1.9M 
28329 0        python /bin/smem -s swap -t        0     6.3M     6.5M     7.4M 
 2719 1001     gnome-pty-helper               16.0K    72.0K    73.0K   516.0K 
  619 0        @sbin/mdadm --monitor --sca    28.0K    72.0K    73.0K   248.0K 

[big snip]

32079 42       gnome-shell --mode=gdm         41.9M     1.9M     2.0M     5.0M 
32403 1001     /opt/google/chrome/chrome -    43.1M   118.5M   119.4M   132.3M 
 4844 1002     /opt/google/chrome/chrome      48.1M    38.1M    41.9M    51.9M 
 5411 1002     /opt/google/chrome/chrome -    54.6M    33.4M    33.5M    36.8M 
 5624 1002     /opt/google/chrome/chrome -    72.4M    54.9M    55.5M    65.7M 
24328 1002     /opt/Adobe/Reader9/Reader/i    77.5M     1.9M     2.0M     5.2M 
 4921 1002     /opt/google/chrome/chrome -   147.2M   258.4M   259.4M   272.0M 
-------------------------------------------------------------------------------
  214 14                                       1.1G     1.1G     1.2G     1.7G 

The source also provides smemcap that will store all the relevant data so that smem can be run on it later.

   To  capture  memory statistics on resource-constrained systems, the the
   smem source includes a utility named  smemcap.   smemcap  captures  all
   /proc entries required by smem and outputs them as an uncompressed .tar
   file to STDOUT.  smem can analyze the output using the --source option.
   smemcap is small and does not require Python.
1
  • 1
    Smem from F17 repo did not work (showed an empty list) but the one from the source worked and is showing almost same numbers as the others :-) mysql 358.1M out of 392.6M total, while top shows 1191224k used Nov 7, 2013 at 0:33
4

You should check this script on another machine, because my system show correct swap usage:

# Your_script.sh
111280   Total Swap Used
# free
Swap:     33551716     120368   33431348

Very nearby 111280 ~= 120368.

Also, look at this script:

for proc in /proc/*; do cat $proc/smaps 2>/dev/null | awk '/Swap/{swap+=$2}END{print swap "\t'readlink $proc/exe'"}'; done | sort -n | awk '{total+=$1}/[0-9]/;END{print total "\tTotal"}'

From this thread:

https://unix.stackexchange.com/questions/71714/linux-total-swap-used-swap-used-by-processes

2
  • The mentioned script is returning same results ... 364920 /usr/libexec/mysqld 400372 Total Nov 5, 2013 at 19:52
  • When I tried the script on other server with very low swap usage (50MB) it reported cca 72MB in total :-) I will need to simulate it on some nonproduction server later... Nov 5, 2013 at 20:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .