I'm having a problem with a virtual hosted server running CentOS. In the past month a process (java based) that had been running fine started having problems getting memory when the JVM was started.

One strange thing I've noticed is that when I start the process, the PID says it is using 470mb of RAM while the 'used' memory immediately drops by over a 1GB. If I run 'top', the total RES used across all processes falls short of the 'used' listed at the top by almost 700mb.

The support person says this means I have a memory leak with my process. I don't know what to believe because I would expect a memory leak to simply waste the memory that the process is allocated- not to consume additional memory that doesn't show up using 'top'.

I'm a developer and not a server guy so I'm appealing to the experts. To me, if the total RES memory doesn't add up to the total 'used' it indicates that something is wrong with my virtual server set-up. Should I suspect a memory leaking java process in this case?

If I use free before:

             total       used       free     shared    buffers     cached  
Mem:       2097152     149264    1947888          0          0          0  
-/+ buffers/cache:     149264    1947888  
Swap:            0          0          0 

free after:

             total       used       free     shared    buffers     cached  
Mem:       2097152    1094116    1003036          0          0          0  
-/+ buffers/cache:    1094116    1003036  
Swap:            0          0          0  

So it looks as though the process is using (or causing to be used) nearly 1GB of RAM. Since the process (based on top is only using 470mb, does that mean that the kernel is all of a sudden using an additional 500mb?

Edit: Added output of cat proc/meminfo - with the java app running:

MemTotal:      2097152 kB
MemFree:       1112672 kB
Buffers:             0 kB
Cached:              0 kB
SwapCached:          0 kB
Active:              0 kB
Inactive:            0 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2097152 kB
LowFree:       1112672 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:               0 kB
Writeback:           0 kB
AnonPages:           0 kB
Mapped:              0 kB
Slab:                0 kB
PageTables:          0 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:         0 kB
Committed_AS:        0 kB
VmallocTotal:        0 kB
VmallocUsed:         0 kB
VmallocChunk:        0 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB
  • Does your Java process use multiple threads? Or does it spawn off one or more child processes? That may explain the discrepancy. If a child process is created and destroyed, the OS may not yet have had a chance to reclaim the memory. Can you tell us the nature of the Java app? Does it open a bunch of files? – Barry Brown Jun 26 '11 at 18:34

The "used" field in top reflects the total amount of RAM being used, including that by processes, file buffers, and cache. But the RES number shows only the amount of RAM being used by a process, exclusive of the file buffers.

To really see how much RAM is being used without the buffers, use the free command and look at the -/+ buffers/cache line. The number in the "used" column shows how much RAM is actually being used by processes.

Here's an example:

$ free
             total       used       free     shared    buffers     cached
Mem:       2057196    1812352     244844          0     344768     833660
-/+ buffers/cache:     633924    1423272
Swap:      2097148          0    2097148

This output shows that the system has 2GB of RAM, of which 1.8GB is being used. However, 344MB of that is used by buffers and 833MB is used by cache. Subtracting that out leaves only 633MB being used by processes and kernel.

Run the free command before and after launching your Java app to get a better idea of what's going on.

Edit: removed references to the kernel memory, which is not reported by these tools.

| improve this answer | |
  • Thanks! That's very helpful. i updated my post above with data from the free command. – rilkeanmind Jun 25 '11 at 4:11
  • Could be. What's the output of cat /proc/meminfo? – Barry Brown Jun 25 '11 at 6:06
  • I added the output. Thanks for fixing my noob formatting btw. – rilkeanmind Jun 25 '11 at 10:40
  • You have a freaky system where most of the numbers are zero. Is there no swap space allocated at all? – Barry Brown Jun 25 '11 at 19:22
  • When I do a top it shows at least one process with 110mb VIRT used. I had assumed that this is swap space. Could this be screwy because it's a virtual hosted server? – rilkeanmind Jun 25 '11 at 22:02

Don't let the cache and swap zeroes worry you since it is a VPS host.

I/O caching and swapping are probably handled by the underlying virtualization host and they probably decided that swapping and caching i/o in guests, with their setup, imposes a performance penalty or something, which might be a good thing since it means that you get 2GB of RAN just for your processes. Usually hosting provider engineers know what they are doing.

This may also be the reason why the things don't quite add up since, due to the way your VPS is jailed (if they use something like OpenVZ) you might not see all things that use up your RAM in top, but are required for your processes to run and are thus counted against your quota.

Also you're not getting the whole picture if all you're comparing against is your one Java process since your app might start additional processes, even bunch of them, that add up. That much memory not being accounted for, tho, could easily be a memory leak of the Java app. It's spawning processes and allocating RAM without returning them back to OS properly, most probably.

Does your app start or use any native process? Does it spawn other processes or threads?

| improve this answer | |
  • That's very helpful. I'm actually using the grails/groovy framework running on the JVM. I am not sure about whether it spawns threads but should be easy to check. At this point I suspect that the problem must be with my app and/or it's interaction with the framework. The puzzling thing is that it worked fine and only recently started having this issue (I hadn't made any code changes to my knowledge). Since I got my answer (leaks are a distinct possibility), I'll close out this question. – rilkeanmind Jun 30 '11 at 13:44
  • I tried to upvote yours but don't have enough reputation yet. – rilkeanmind Jun 30 '11 at 13:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.