2

We have a CentOS 7 server that goes into swap a little bit about once a week, despite having > 20GB of available memory, and swappiness is set to 0. We want to be able to use swap as a sort of sentinel that we are running out of memory without having to worry about triggering the OOM killer.

What else might be causing the system to go into swap, and what besides swappiness can we do to prevent it from swapping?

  • If your question is more about "What causes a CentOS Linux host to swap?" you might have more success looking at the questions on unix.stackexchange.com , including this one. unix.stackexchange.com/questions/224958/… Good Luck! – StandardEyre Oct 31 '16 at 17:49
  • If you are on a NUMA hardware, swappiness can also depend on the usage of a single NUMA node. Here is an example of NUMA and swap based on MySQL, but it also applies to applications which use memory in the same way. You also shouldn't set swappiness to 0, because the system might start to buffer out and buffer in shared libraries. – Thomas Oct 31 '16 at 19:03
  • @Thomas This is a NUMA machine, but no processes are allocating a huge amount of memory, 1GB is the most that one process will use and it's usually lower than that, and we have 32GB of memory, of which 20 is free. – Jay Paroline Oct 31 '16 at 19:29
2

Swapping isn't an indication that you're running out of memory. It's an indication that memory pages are being moved.

When moving memory regions for various purposes (be it defragmentation, idle page offloading, or optimization of some other sort), the easiest and most effective way to do that is to to put it into swap, and then re-allocate it into primary memory when (or typically before) it's actually needed. This process can be forced by some applications to solve problems internally, such as extreme memory fragmentation. This will happen despite your swappiness setting.

Turning swappiness to zero only makes sure that instead of being able to proactively send idle pages down to swap, everything is sent down to it at once during a critical OOM event. Swap typically isn't able to write quickly enough to solve this problem, and then the OOMkiller goes and starts shooting processes in the head. The logic it uses for this is somewhat aggressive, and it'll probably end up trying to kill init if things are busy.

So instead of increasing performance, this will make sure that your machine crashes if it runs out of memory (even if it's almost all idle pages that are doing nothing), regardless of how much swap you actually have. It will also increase memory fragmentation, reducing performance and stability in many scenarios.

If you don't want a machine to ever swap or page, don't put swap on it. This is a bad idea in most scenarios, but can be appropriate for diskless systems and SOME virtualized workloads.

If you want to limit swapping in a system to potentially decrease application latencies, set swappiness to around 10. This is for systems with an excess of memory, such as yours.

If you want to know if your machine is running out of memory, point some actual monitoring at it.

| improve this answer | |
  • I have actual monitoring, thanks, and this doesn't answer the question about how to track down why my server is going into swap in this instance. Running this same setup on CentOS 6, I did not encounter this behavior, and in fact I still don't on most of my CentOS 7 servers. – Jay Paroline Oct 31 '16 at 22:26
  • What is your workload (and associated versions), and what exact version of the kernel are you using? – Spooler Oct 31 '16 at 22:31
  • 3.10.0-229.14.1.el7.x86_64, nginx/1.6.3, PHP 5.4.16 – Jay Paroline Nov 1 '16 at 2:44
0

just because you have swap allocated doesn't mean that swapping IN/OUT is occurring. Check vmstat to see if swapping is occurring. and since you have swappiness set to 0 it is more then likely an empty swap.

look at values of si and so for in/out in the vmstat utility.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.