Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am having a really odd problem and I am lost a bit as I do not know what to do any more.

We are running in production 4 dedicated memcached boxes. All boxes have 48Gb of RAM, they are running memcached and nothing else and the daemon's memory limit is set to 42Gb.

The problem is that no matter the amount of traffic and gets/sets the boxes receive the cache will fill up on all 4 of them to about 38Gb but then the amount of free RAM available to the operating system will start slowly dropping over the course of several days until the boxes will start swapping, filling up the swap and thrashing! Now this is really strange as there is nothing else running on the boxes that could fill up the rest of the RAM and memcached is eating up 38Gb and not growing (at least that is what the graphs and the stats show).

I have tried setting the swappiness to 0 but it didn't help. I have tried lowering the cache limit even more but I get the same behavior.

I am running Centos 5.6, 2.6.18-238, memcached 1.4.4 and libevent-1.4.13-1.

Have any of you ran into a similar problem before? Could memcached possibly be leaking memory and not showing up in the graphs or the usual Linux tools?

Thanks! Dan

share|improve this question
add comment

2 Answers

up vote 0 down vote accepted

Memcached could leak memory (these things happen), but it'd show up in the memory accounting. Absent a fairly unlikely kernel bug, memory accounting will be accurate. The long and the short of it is that you're missing something in your diagnostic actvities. Collect more data and keep staring at it.

share|improve this answer
    
Thanks for the advice! It does sound reasonable. I was thinking that I was missing something too, but I wanted to make sure it is not something obvious in memcached's behavior that I am missing. –  danakim Jul 21 '11 at 15:59
add comment

First things first: is it really necessary for you to have such high memory limit for memcached? Would less than 42 GB be enough in practise?

share|improve this answer
    
Well, in practice I could go for less but our traffic and hence our SQL queries will grow soon and I want to make sure I have enough capacity. Plus, if I have 48Gb of RAM it just feels like a waste not to use as much as I can. –  danakim Jul 21 '11 at 16:01
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.