1

One of our sap systems(PI ABAP+JAVA stack) was giving performance issue. The entire 64GB configured for the machine gets hogged up(and the 8 cores as well). Every one is suspecting the java part,but i think different.

The java server nodes where getting restarted with Out Of Memory error. Looking at the hprof files, i found that they where only 1.2G(avg of 3 server nodes) in size , when 3GB(both -Xms and Xmx) of heap is configured for the server nodes. This observation lead to the following doubt.

I have read that when Xms and Xmx are set to the same value, the jvm is allocated the entire heap when it starts. If its the case the server nodes would have 3GB of heap from the start. If so why does it doesn't reflect in the hprof file or if the hprof contains only the memory allocated to objects during runtime, the the size clearly idicates that the heap memory was free(more than 50%), so how OOM error...!!..??

I also know that linux does something called as memory over-commit. ie memory is not actually given when its requested but when its actually used. Is this contributing to the out of memory exception. Like when the JVM starts the os says to it that you have been allocated 3GB of memory but actually defers it until its actually required. By the time the jvm actually tries to allocate the memory to objects , some other applications might have exhausted the memory. Is this possible...??

Even if the java nodes had memory leak issue, wouldn't it be confined to the 3GB of heap. How can it hog up the entire 64G of physical memory....???

One more thing i observed was the swap space was only 50% used.

Any light on this...!

2 Answers 2

1

SAP OSS was also looking into the issue. Today i got the reply from them. My observation was correct. Java was not the culprit. ABAP stack was facing some issue and not releasing memory. After restarting ABAP work process, memory got released at OS level.

But i would also like to get an understanding on the highlighted portion of the question, like whether such a situation can occur or not, resulting in JAVA OOM errors...??..!!. Any information in this regard will be helpful.

0

Overcommit is by default enabled on Linux in heuristic mode. That means that kernel will usually allow overcommit - meaning that will promise more memory to all of the processes requesting it then it actually can deliver, in a hope that processes will never actually start using all the memory at the same time. Maybe overcommit is disabled on your server, you can check it by running:

$ cat /proc/sys/vm/overcommit_memory

If the value is 0, heuristic overcommit is turned on.

If the situation occurs that actual memory usage grows over the amount of RAM that system can provide, kernel will activate OOM killer which will try to kill processes to free memory. It will usually kill the youngest processes consuming large amounts of RAM, but you cannot depend on it. It can (and will) cause havoc. You can modify affinity of OOM to kill specific processes by adjusting /proc//oom_adj (for example if you want to avoid situation where OOM kills database or some other large RAM [ab]user).

So, if your system enters OOM phase, consequences for Java processes could be that they get killed instantaneously - which wouldn't get you 'Out of memory' messages in java logs you are observing.

Setting both Xmx and Xms to same value will prevent heap resize, but that doesn't mean java process will start using all the memory all at once at the startup. It will allocate as much as it needs VIRT memory, but resident data set won't grow up to Xms but will stay as low as needed.

In terms of virtual memory: kernel will promise (overcommit) to java process as much as it asks for (Xmx + some additional), but all that memory won't get allocated immediately. Amount needed for current data will be allocated only, and you can see how much by observing resident set size (non-swapped physical memory a task has used). To see the VIRT and RSS sizes you can run the following command:

$ ps aux | egrep '(^USER|java)'
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
tomcat   10229 21.5  9.1 6813688 548344 ?      Sl   09:01   1:10 ....java...

By all probability, errors you are observing are indication that program running under java virtual machine process lacks heap space. Try by increasing the Xmx setting and re-test your app.

3
  • hai, as said in the question , the jvm newer used the configured heap space. We have wily introscope configured as well for monitoring the JVM. It also shows that the heap memory was never fully utilised. So how can it be a heap memory issue...!!..??
    – varun
    Dec 23, 2014 at 11:47
  • How much of it was used comparing to free mem on system? Dec 23, 2014 at 23:47
  • It was just using the configured amount only and it was more than enough for the server nodes. Total around 23% of the total physical memory configured.
    – varun
    Dec 24, 2014 at 4:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .