Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am trying to do some tests with Azul's ZingVM, which needs to pre-allocate large amount of ram.

The problem is, on my server (running Cent OS 5.8) which has 16GB ram, more than 8GB will be used right after rebooting... like this (I reboot the server, ssh login to it, and run free -m at once):

             total       used       free     shared    buffers     cached
Mem:         16048       8281       7767          0          1         19
-/+ buffers/cache:       8260       7788
Swap:        18047          0      18047

Obviously the buffers and cached don't contribute to the used 8GB ram. This is an application server which runs Apache Tomcat everyday (for tests), and I am sure that the Tomcat won't be started automatically.

I tried to find out which process is using such a large amount of memory with top, but only to find that the top-RES-using process only uses 4460k...

However I have another server (with Cent OS 5.8) running mysql. The memory usage of this database sever is normal. Now I am testing ZVM on the database server, and with Zing allocating 65% ram on startup, after a fresh rebooting, the free -m shows:

             total       used       free     shared    buffers     cached
Mem:         16048      10942       5105          0          3         55
-/+ buffers/cache:      10883       5165
Swap:        18047          0      18047

So... If we ignore the 65% of 16GB -- about 10GB -- used by Zing, the used memory after rebooting should be just several MBs.

Now I am just testing if I can get Tomcat working under ZingVM. Then I need to run real tests on another two Cent OS servers, and unfortunately, those two servers have the same problem. That is to say, we have 4 servers, and 3 servers have this problem.

What may be the reason of this problem? Can I resolve it without re-installing the OS on those machines?


Update:

here is what zing-system-tool reports when I try to set it to use 75%(it's default value) of the memory. Exactly the same pattern of output when I specify 85%.

Fatal error:  Not enough free memory (memoryForAzulBytes (12620660736) > MemFree (8142327808)).
zing-memory: ERROR: Configure pmem failed
Fatal error:  os::safeSystem:  command '/sbin/service zing-memory restart' failed.

And here is a successful allocation on the database server (I specified 68% this time, and the old value is 65%; ZST will release the memory reserved by itself):

Info:  azulPmemPages: 5456.
INFO: az_pmem_reserve_pages (num2mPages 5456) succeeded
INFO: az_pmem_fund_transfer (to 7, from 0, bytes 11442061312) succeeded
INFO: az_pmem_fund_transfer (to 1, from 7, bytes 570425344) succeeded
INFO: az_pmem_fund_transfer (to 3, from 7, bytes 570425344) succeeded
INFO: az_pmem_fund_transfer (to 0, from 7, bytes 10301210624) succeeded
Info:  You can now start Java processes up to -Xmx9824m or -Xmx9g.
Info:  Azul pmem initialized successfully.
zing-memory: INFO: start successful
zing-memory: INFO: restart successful

Update 2:

I saw this in /proc/meminfo on the problematic server:

HugePages_Total:  4096
HugePages_Free:   4096
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

while on the database server, it's all 0. Could this be the reason of the used memory?

share|improve this question
2  
RAM being in use is not a problem. If you don't want your RAM to be used, take it out of the machine. It's not like if you use only 8GB right now, you can use 24GB later. Any RAM not used now is forever wasted. –  David Schwartz Mar 28 '12 at 8:39
    
@DavidSchwartz I need to allocate 85% of the 16GB for ZingVM and it's impossible because 8GB has been used. Also, I've read a lot of pages stating that the used memory is not a problem -- but that just applies to the situation that Linux uses memory for cache. –  coolcfan Mar 28 '12 at 8:44
    
What precisely happens when you attempt to allocate the memory? –  David Schwartz Mar 28 '12 at 8:54
    
@DavidSchwartz I will add the output in the question. –  coolcfan Mar 28 '12 at 8:56
add comment

2 Answers

OK, I figured it out:

See my update 2 in the question. That is the reason.

Hugepage is reserved; unless specified, applications won't use it. So ZST can't allocate the memory.

When I set Hugepage to zero, the memory is released at once, and ZST allocates memory successfully.

share|improve this answer
    
Upvoted other answers and comments. –  coolcfan Mar 28 '12 at 9:44
add comment

Try slabtop on the problematic server and see if kernel allocates huge amounts of RAM for some reason. For example XFS can allocate lots of memory for its cache without being visible at buffers/cached columns at all (though XFS will release the memory when needed).

share|improve this answer
    
I suspect that on the problematic server there is also something like Zing System Tool which reserves memory upon startup. On the only normal server, ZST allocates 10GB ram, and I can't see any memory-eating process from top. For slabtop, the total size in the output is around 14000K on the problematic server, and 156xxK on the database server. –  coolcfan Mar 28 '12 at 8:49
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.