3

I have several high memory linux servers running centos 64bit, and after 10-20 days of uptime I have noticed on all of these servers they dont actually use all the ram available to then (they all have around 10gb free on a 48gb system ,and 20gb free on 64gb systems.

They are web servers, and have a working dataset (Eg active files ) of over the amount of ram in the server so I would have assumed that the page cache would grow to the point of using all ram then the pages from the page cache would be freed when / if needed.

for example :

top - 09:44:46 up 57 days,  9:32,  5 users,  load average: 6.44, 6.33, 6.27
Tasks: 680 total,   4 running, 676 sleeping,   0 stopped,   0 zombie
Cpu(s): 17.3%us,  3.3%sy,  0.0%ni, 79.0%id,  0.1%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:  49313212k total, 39277524k used, 10035688k free,  1247064k buffers
Swap: 20643832k total,        0k used, 20643832k free, 20592968k cached

shows this server has been up for 57 days, however has 10GB of ram free that should be used in the page cache.

The following sysctl's are set from stock centos :

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv4.conf.all.log_martians = 1
error: "kernel.maps_protect" is an unknown key
net.core.rmem_default = 8388608
net.core.wmem_default = 8388608
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_max_syn_backlog = 4096
net.core.netdev_max_backlog = 20000
vm.min_free_kbytes = 85536
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_syn_backlog = 6000
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_fin_timeout = 40
net.ipv4.tcp_keepalive_time = 1000
net.ipv4.tcp_keepalive_probes = 2
net.ipv4.tcp_keepalive_intvl = 30
net.netfilter.nf_conntrack_tcp_timeout_established = 2000
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1
kernel.panic = 40
fs.proc_can_see_other_uid = 0
net.ipv4.ipfrag_secret_interval = 6000
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_max_tw_buckets_ub = 720000
net.core.optmem_max = 25165824
fs.proc_can_see_other_uid = 0
fs.proc_super_gid = 32010
vm.swappiness = 2
fs.file-max = 400000
fs.suid_dumpable = 1
kernel.msgmni = 1024
kernel.sem = 250 256000 32 1024

iv tried altering now one to many sysctl's ! , and hope some kind soul may have seen this before, thanks in advance for any help.

5
  • 1
    It could well be that you simply don't need all that memory...
    – HBruijn
    Nov 24, 2013 at 14:53
  • Yeah. sending around sys config is useless - start finding out which of your processes uses how much, then validate that one by one with hte limits for the process (i.e. database, web server). This likely is no system level problem but - if there is a problem - it is an application level memory that stops an app to use more memory.
    – TomTom
    Nov 24, 2013 at 15:00
  • Are you absolutely certain that your web app is actually reading as much data as you think it is? Nov 24, 2013 at 17:53
  • Yes, you are showing 0.1%wa, so it's not like you are wasting time hitting the disk, if you want to get more work out of these systems, have a look at a good book on tuning, and figure out where the bottleneck is, because it's not the disk. Nov 24, 2013 at 18:45
  • Can you tell us which operating system versions were in use? If this is EL6, the kernel defaults changed over time.
    – ewwhite
    Feb 8, 2014 at 6:32

3 Answers 3

6

Im coming back here to answer my own question!

The issue is caused when you have multi cpu's (not multi core) eg 2 cpus each with x cores

vm.zone_reclaim_mode=0

fixes this.

2
  • You should finish registering, and have your two accounts merged. Jan 31, 2014 at 14:48
  • Interesting. I've got a server that's a bit overspecced for its job, not even using 25% of its total RAM. It might be interesting to see if that parameter has any effect.
    – Paul Gear
    Feb 8, 2014 at 10:28
0

I'm not sure there's a problem here. Are you experiencing other issues like high disk IO?

Most software isn't designed to consume all memory just because it can. You don't mention what web server software you're using, but most web server software will have internal algorithms designed to flush objects from cache if they're not requested often enough. They also have configurable mechanisms designed to control the amount of resources they consume on the server.

Assuming you're using Apache, the performance tuning section of the Apache documentation has a section about tuning MaxRequestWorkers:

You can, and should, control the MaxRequestWorkers setting so that your server does not spawn so many children it starts swapping. This procedure for doing this is simple: determine the size of your average Apache process, by looking at your process list via a tool such as top, and divide this into your total available memory, leaving some room for other processes.

If you're using Apache and if you've done this, then its likely that you don't have enough requests coming in for Apache to justify maintaining all those workers. It could also be that your estimation of the number of workers is too low and you could increase MaxRequestWorkers to try to get Apache to spawn more workers.

Unless you're experiencing some other issue, it doesn't sound like anything is wrong. But in any case, this is probably a web server software config question/issue and not a sysctl one.

1
  • It might have been that there wasn't a problem, except for the fact that @Kev pointed out that his working set of files was greater than the amount of RAM in the server. Therefore it all should have been used as cache.
    – Paul Gear
    Feb 10, 2014 at 5:51
-2

Any RAM not used for programs is used as a disk cache. So, in theory, all RAM is used. A great explanation is here: http://www.linuxatemyram.com/

1
  • That doesn't answer the question. In fact, if @Kev didn't already know that, he wouldn't ever have asked the question.
    – Paul Gear
    Feb 8, 2014 at 10:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .