I have a 24-core server with 64G of DRAM, and it's running Ubuntu Server 15.10. I am getting some strange behavior with respect to swapping and physical memory use. I never had this trouble until I upgraded Ubuntu to 15.10.
You can see my problem in 'top':
top - 18:52:09 up 1 day, 2:25, 3 users, load average: 1.64, 1.30, 1.18 Tasks: 525 total, 2 running, 523 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.3 us, 1.3 sy, 0.0 ni, 97.0 id, 1.4 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 65937528 total, 37526160 used, 28411368 free, 14396 buffers KiB Swap: 67071996 total, 67071724 used, 272 free. 104304 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 363 root 20 0 0 0 0 R 100.0 0.0 324:02.98 kswapd0 6725 theosib 20 0 99.398g 0.034t 8920 D 12.0 55.1 59:08.71 common_shell_ex
There's a single user logged in (me), and I have a single process using a large amount of virtual memory. However, something is limiting it to around half the physical memory, while the swap partition is basically full. I've watched these processes (Synopsys Design Compiler) run, and they don't break the 50% mark until swap fills. And another weird thing is that kswapd0 uses very high (usually 100%) CPU. AFAIK, kswapd0 should be I/O bound and therefore not use a lot of CPU time.
I've looked to see if there were any limits being imposed, but ulimit says otherwise:
$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 257447 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 257447 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
I've also tried setting swappiness to 10, but that didn't help.
Can anyone help me figure out why this system is misbehaving?