I am trying to do some tests with Azul's ZingVM, which needs to pre-allocate large amount of ram.
The problem is, on my server (running Cent OS 5.8) which has 16GB ram, more than 8GB will be used right after rebooting... like this (I reboot the server, ssh login to it, and run
free -m at once):
total used free shared buffers cached Mem: 16048 8281 7767 0 1 19 -/+ buffers/cache: 8260 7788 Swap: 18047 0 18047
Obviously the buffers and cached don't contribute to the used 8GB ram. This is an application server which runs Apache Tomcat everyday (for tests), and I am sure that the Tomcat won't be started automatically.
I tried to find out which process is using such a large amount of memory with
top, but only to find that the top-RES-using process only uses 4460k...
However I have another server (with Cent OS 5.8) running mysql. The memory usage of this database sever is normal. Now I am testing ZVM on the database server, and with Zing allocating 65% ram on startup, after a fresh rebooting, the
free -m shows:
total used free shared buffers cached Mem: 16048 10942 5105 0 3 55 -/+ buffers/cache: 10883 5165 Swap: 18047 0 18047
So... If we ignore the 65% of 16GB -- about 10GB -- used by Zing, the used memory after rebooting should be just several MBs.
Now I am just testing if I can get Tomcat working under ZingVM. Then I need to run real tests on another two Cent OS servers, and unfortunately, those two servers have the same problem. That is to say, we have 4 servers, and 3 servers have this problem.
What may be the reason of this problem? Can I resolve it without re-installing the OS on those machines?
here is what zing-system-tool reports when I try to set it to use 75%(it's default value) of the memory. Exactly the same pattern of output when I specify 85%.
Fatal error: Not enough free memory (memoryForAzulBytes (12620660736) > MemFree (8142327808)). zing-memory: ERROR: Configure pmem failed Fatal error: os::safeSystem: command '/sbin/service zing-memory restart' failed.
And here is a successful allocation on the database server (I specified 68% this time, and the old value is 65%; ZST will release the memory reserved by itself):
Info: azulPmemPages: 5456. INFO: az_pmem_reserve_pages (num2mPages 5456) succeeded INFO: az_pmem_fund_transfer (to 7, from 0, bytes 11442061312) succeeded INFO: az_pmem_fund_transfer (to 1, from 7, bytes 570425344) succeeded INFO: az_pmem_fund_transfer (to 3, from 7, bytes 570425344) succeeded INFO: az_pmem_fund_transfer (to 0, from 7, bytes 10301210624) succeeded Info: You can now start Java processes up to -Xmx9824m or -Xmx9g. Info: Azul pmem initialized successfully. zing-memory: INFO: start successful zing-memory: INFO: restart successful
I saw this in /proc/meminfo on the problematic server:
HugePages_Total: 4096 HugePages_Free: 4096 HugePages_Rsvd: 0 Hugepagesize: 2048 kB
while on the database server, it's all 0. Could this be the reason of the used memory?