Linux out of memory on VPS

Question

On my VPS (CentOS) 4GB ram (2 + 2 dynamically assigned when needed) and i have tomcat running with following options JAVA_OPTS="-Xms256m -Xmx2048m -XX:MaxPermSize=256m".

Now when i try to start other java based app (like hudson server) i get following error:

 There is insufficient memory for the Java Runtime Environment to continue.
 Native memory allocation (malloc) failed to allocate 664080 bytes for Chunk::new
 An error report file with more information is saved as:
 /server/hs_err_pid26476.log

It seems as if system could not allocate 650KB of memory, but it should have like 1.2 GB free.

free -m 
             total       used       free     shared    buffers     cached
Mem:          4096       2816       1279          0          0          0
-/+ buffers/cache:       2816       1279
Swap:            0          0          0

I found somewhere that i should also check /proc/user_beancounters, which shows failcnt on privvmpages. I have no clue what that means (does the provider give 4GB of memory as he is supposed to? or is he cheating?)

# cat /proc/user_beancounters
Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
 70692271:  kmemsize                 15371949             15388993             41943040             46137344                    0
            lockedpages                     0                    0                 1024                 1024                    0
            privvmpages                720030               720189              1048576              1048576                 9604
            shmpages                     9001                 9001                65536                65536                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numproc                       109                  109                  256                  256                    0
            physpages                  173670               173780  9223372036854775807  9223372036854775807                    0
            vmguarpages                     0                    0               262144               262144                    0
            oomguarpages               174598               174708  9223372036854775807               262144                    0
            numtcpsock                     30                   30                 1440                 1440                    0
            numflock                       14                   14                  752                  826                    0
            numpty                          1                    1                   64                   64                    0
            numsiginfo                      0                    0                 1024                 1024                    0
            tcpsndbuf                  601840               601840              6881280             10813440                    0
            tcprcvbuf                  491520               491520              6881280             10813440                    0
            othersockbuf               174600               174600              4504320              8388608                    0
            dgramrcvbuf                     0                    0              1048576              1153432                    0
            numothersock                  117                  118                 1440                 1440                    0
            dcachesize                      0                    0              7340032              8074035                    0
            numfile                      3516                 3516                16384                16384                    0
            dummy                           0                    0                    0                    0                    0
            dummy                           0                    0                    0                    0                    0
            dummy                           0                    0                    0                    0                    0
            numiptent                      18                   18                  400                  405                    0

Answer 1

OpenVZ providers typically sell RAM capacity of their virtual machines as two numbers: "guaranteed" and "burst" RAM. You are supposed to always get the "guaranteed" amount of RAM, and may use up to the "burst" amount if host resources permit. For example a VPS might be sold as "512MiB guaranteed 1GiB burst RAM".

You should always have a successful memory allocation below the "guaranteed" amount, but memory allocations may sometimes fail above that amount, if other virtual machines on the host are consuming RAM as well, or if the provider has oversubscribed the machine (almost always).

To learn what these numbers are, look at the barrier for vmguarpages which represents the "guaranteed" RAM, and the limit for privvmpages which represents the "burst" RAM. Each of these numbers is expressed in a number of 4KiB pages.

So in your case, we can see that you have 1GiB (262144) "guaranteed" RAM, and 4GiB (1048576) "burst" RAM. (If these aren't the numbers you were promised, you need to have a very unpleasant talk with your VPS provider.)

Since you are already well over 2GiB used in the VM, your memory allocation may fail, because it is above the "guaranteed" amount.

Answer 2

Your provider very likely only guarantees 2 GB of memory and two more if available, that is, if the other costumers currently need less (which rarely will be the case). You would hence be better off to assume a maximum of 2 rather than 4 GB.

The failcount on privvmpages shows how often your system failed to allocate some memory (roughly speaking).