I have virtual linux (Fedora 17) server with 28GB RAM and 2GB swap. The server is running mysql db that is set up to use most of the RAM.
After some time running the server starts to use swap to swap out unsued pages. That is fine as my swappiness is at default 60 and it is the expected behaviour.
The strange thing is that the number in top/meminfo does not correspond with info from processes. I.e. the server is reporting these numbers:
/proc/meminfo: SwapCached: 24588 kB SwapTotal: 2097148 kB SwapFree: 865912 kB top: Mem: 28189800k total, 27583776k used, 606024k free, 163452k buffers Swap: 2097148k total, 1231512k used, 865636k free, 6554356k cached
If I use the script from http://serverfault.com/a/423603/98204 it reports reasonable numbers (few MBs swapped by bash'es, systemd, etc) and one big allocation from mysql: (I ommited a lot of output lines)
892  qmgr -l -t fifo -u 896  /usr/libexec/postfix/master 904  mysql -u root 976  -bash 984  -bash 992  SCREEN 1000  /bin/bash 1192  sshd: admin@pts/0 1196  sshd: admin [priv] 1244  /usr/lib/systemd/systemd 9444  /usr/bin/perl /bin/innotop 413852  /usr/libexec/mysqld --basedir=/usr --datadir=/data/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/data/mysql/err --open-files-limit=8192 --pid-file=/data/mysql/pid --socket=/data/mysql/mysql.sock --port=3306 449264 Total Swap Used
So if I get the script output right the total swap usage should be 449264K=cca 440MB with mysql using cca 90% of the swap.
The question is why this differs so much from the top and meminfo numbers? Is there any way how to "dump" swap info to see what is actually in it instead of summing the swap usages from all processes?
When analyzing the issue I came up with different ideas but they all seems to be wrong:
- The script output is not in KB. Even if it would be in 512 or 4KB units it won't match. Actually the ratio (1200:440) is about 3:1 which is "strange" number.
- There are some pages in swap that are somehow shared between processes as mentioned in http://serverfault.com/a/477664/98204 . If this is true how can I find the actual number of memory used like this? I mean it would need to make cca 800MB difference. And that does not sound right in this scenario.
- There are some "old" pages in swap used by processes that already finished. I would not mind that if I were able to find out how much is this "freeable" swap.
- There are pages in swap that has been swapped back to memory and are in swap just in case they did not change in RAM and need to be swapped out again as mentioned in http://serverfault.com/a/100636/98204 . But the SwapCached value is only 24MB.
The strange thing is that the swap usage is slowly increasing while the sum output from the script is roughly the same. In last 3 days the swap used increased from 1100MB to current 1230MB while the sum increased from 430MB to current 449MB (cca).
The server has enoough free(able) RAM so I could just turn off the swap and turn it back on.Or I could probably set swappiness to 0 so the swap would get used only if the is no other way. But I would like to solve the issue or at least find out what is the cause of this.