Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 8 down vote favorite
2

Updated, see bottom of the longish (sorry) question.

Looking at our memcached stats I think I have found an issue I was not aware of before. It seems that we have a strangely high amount of wasted space. I checked with phpmemcacheadmin for a change, and found this image staring at me:

memcached cache size graphic

Now I was under the impression that the worst-case scenario would be that there is 50% waste, although I am the first to admit not knowing all the details. I have read - amongst others- this page which is indeed somewhat old, but so is our version of memcached. I think I do understand how the system works (e.g.) I believe, but I have a hard time understanding how we could get to 76% wasted space.

The eviction rate that phpmemcacheadmin shows is 2 ev/s, so there is some problem here.

  • The primary question is: what can I do to fix this. I could throw more memory at it (there is some extra available I think), maybe I should fiddle with the slab config (is that even possible with this version?), maybe there are other options? Upgrading the memcached version is not a quickly available option.

  • The secondairy question, out of curiosity, is of course if the rate of 75% (and rising) wasted space is expected, and if so, why.

System: This is currently not something I can do anything about, I know the memcached version isn't the newest, but these are the cards I've been dealt.

  • Memcached 1.4.5
  • Apache 2.2.17
  • PHP 5.3.5

As a response to @DavidSchwartz 's answer: here are the slab statistics that phpmemcacheadmin produces: (there are more slabs btw then these)

(I have also pasted stats from a bit later in text format here)

Slab details

UPDATE

I've restarted the daemon with -f 1.5, and it looked really good. After some warming we had a used / wasted of 50 / 50 . But, the same as before, the longer we had in the day (it gets busier during the day) it started falling back to what it currently is: 30 / 70, and wasted is still rising. Apart from that, I still don't know where the 'wasted' is coming from. I see this slab:

**Slab 5 Stats**
Chunk Size  496.0 Bytes
Used Chunk  77502 [24.6 %]
Total Chunk 314986
Total Page  149
Wasted      117.3 MBytes
Hits        30.9 Request/sec
Evicted     0

It isn't full, it has no evicted, but it is wasting 117.3 MBytes. The quick calculation I did (correct me if I'm wrong) was:

  • the previous slab has a chunk size of 328, so worst case this slab is filled with 329 byte chunks.
  • this means that it is wasting 167 bytes per used chunk = 12942834 bytes = 12.3 MB

So then where did the other 105 MB wasted come from? It's bigger brother right next to it looks like this:

**Slab 6 Stats** 
Chunk Size  744.0 Bytes
Used Chunk  17488 [31.0 %]
Total Chunk 56360
Total Page  40
Wasted      31.1 MBytes
Hits        107.7 Request/sec
Evicted     1109
share|improve this question
The problem is that there's tons of unused space in the other slabs, yet slab 3 is 100% full and seeing evictions. – David Schwartz Jun 11 '12 at 11:52
Good point, that would explain the evictions, though I'm not really sure how the 'wasted' number is calculated. If slab 8 only has 13.9% used, there surely must be some "free" space left there? – Nanne Jun 11 '12 at 12:03
Yes, there's free space in that slab. But that doesn't help if the objects that are being evicted don't go in that slab. – David Schwartz Jun 11 '12 at 12:13
That's what I figured from your answer, but why is there no free space listed? There should be some part of that pie-chart white (as it is on my test install) if there is still space left, at least, that's what I figured – Nanne Jun 11 '12 at 12:16

2 Answers

up vote 5 down vote accepted
+50

You probably have a very large number of very small objects. Typically, the smallest slab holds 104-byte entries. If you have a lot of entries that just map one integer to another, you can get waste as high as 85%.

You can find information on how to tune around this in the article Memcached for small objects.

share|improve this answer
If I read the stats page correctly this is not the case. Most waste is in a slab with 480.0 Byte-chunks. Let me check If I can show some stats... – Nanne Jun 11 '12 at 11:00
Oh, then this is fine and normal, nothing to worry about. There's just less data in there now. (Notice, for example, that that slab is only 14% used.) – David Schwartz Jun 11 '12 at 11:39
But how is 75% wasted normal? Does this number include unused space? I'd expect that to be counted as "free". Also, we see an increase in wasted // a decrease in used memory as the day goes by, while the site gets busier. That, and the fact that we do have evictions, make me wonder what can be done. – Nanne Jun 11 '12 at 11:46
Having fewer slabs may help avoid the problem of too much memory getting stuck in the wrong slab. For example, -f 1.5 -I 2800 may help. – David Schwartz Jun 11 '12 at 11:59
The man-page is isn't too clear: the -I 2800, that means 2800K, as opposed to the 1M default? – Nanne Jun 11 '12 at 12:05
show 11 more comments
up vote -1 down vote

I had this issue and moved from memcached to redis (without disk based saving). I know this might not be possible but you could try it as an option and keep an eye on the memory fragmentation. You could even turn persistence on to fix "old cache" issues on restart.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.