2

For some reason our system starts swapping (actively used memory) at around 60GB used memory. (see edit below, it seems I/O and disk cache usage, even of previously run processes has an effect) Turning it off (swapoff-a) for test lead to bad_alloc's once (i guess because there were more processes also using memory at the time) but has also worked to speed up my program by a factor of mroe than 10.

This reproduces the problem (without other significant processes running EDIT: see below it only happens if an I/O intense process ran shortly before):

#include <cstdio>
#include <vector>

int main() {
  size_t bytes = size_t(80) * 1024 * 1024 * 1024; // 80GB
  size_t* data = new size_t[bytes / sizeof(size_t)];
  for (size_t i = 0; i < bytes / sizeof(size_t); ++i) {
    data[i] = i;
  }
  for (;;) {}
}

At around 60GB used memory, the system starts swapping and CPU usage goes below 100% (because the process is I/O bound now, I guess).

System is a Ubuntu 14.04, 64bit:

Linux ... 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Once the point is reached, this what free -m gives me:

              total        used        free      shared  buff/cache   available
Mem:          96671       55504         358          60       40808       40478
Swap:         47679       19366       28313

The problem persists with swappiness=1 (if something I/O intensive has just run and especially for my actual culprit (not the code above) that does both, lots of I/O and use much memory. If there has been little to no I/O recently, the program above allocates all memory and doesn't swap!

The problem goes away if there has been nothing I/O intense for a longer time. It seems the OS somehow makes my application swap because it thinks the disk cache is more valuable - even at very low swappiness. I don't understand this behavior because disk-cache memory should be just as good as free memory and most certainly don't trigger swapping on the running process.

Initially my problem happened in an application that reads large files and uses much memory. Afterwards it persist to the example code above that has no I/O at all. Finally, when I start the example code later on, no swapping occurs.

  • Honestly, I am not sure what is going on. You may need someone who really knows memory management. I suspect you might have better luck if we migrated this over to unix.stackexchange.com. – Zoredache Oct 24 '16 at 22:59
  • @Zoredache how can I (we?) migrate this, then? PS: I edited the question once more. The problem only seems to occur if there is (or recently has been) something I/O intense. Still, swapoff -a tremendously boosts my performance, and as I see it, this shouldn't be the case. – b.buchhold Oct 25 '16 at 12:27
  • The OS is 64 bit, but does you compiled you app in 64 bit or some flag is there to make it 32 bit ? (CFLAGS=-m32) – yagmoth555 Oct 25 '16 at 13:00
  • Do you actually run processes that request multiple tens of gigabytes of memory at once? – Michael Hampton Oct 27 '16 at 7:07
0

64 GB seems to be the limit for 32 bit Ubuntu with PAE support: https://help.ubuntu.com/community/32bit_and_64bit

A 32-bit computer has a word size of 32 bits, this limits the memory theoretically to 4GB. This barrier has been extended through the use of 'Physical Address Extension' (or PAE) which increases the limit to 64GB although the memory access above 4GB will be slightly slower.

  • it's a 64bit system. is there any possibility unnecessary PAE is still active and causing problems? I didn't set up the system myself – b.buchhold Oct 24 '16 at 15:23
  • grep --color=always -i PAE /proc/cpuinfo If it outputs something, you have PAE support – Michal Oct 24 '16 at 15:29
  • so there is lots of pae support. could this be harmful on a 64bit system? – b.buchhold Oct 24 '16 at 17:49
  • it's not harmful it's just fact it allocates everything which overpass 60 gb so you need to turn it off to get all of your memory being available for system (not to swap unless there's need). – Michal Oct 25 '16 at 7:06
0

Check vmstat to see the si, so (swap in/out) is actually populating any swap. even if swappiness is set to 0 swap can allocate swap.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.