Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm going nuts trying to scale memcached. From their site:

Memcached operations are almost all O(1). Connecting to it and issuing a get or stat command should never lag. If connecting lags, you may be hitting the max connections limit. See ServerMaint for details on stats to monitor.

If issuing commands lags, you can have a number of tuning problems. Most common are hardware problems, not enough RAM (swapping), network problems (bandwidth, dropped packets, half-duplex connections). On rare occasion OS bugs or memcached bugs can contribute.

Well.. it is most certainly not performing like an O(1) operation for me. Under low to normal load on our site memcached response times for get and set ops are about 0.001 seconds. Not bad. But if we triple the load we get outliers that take 100x (or in rare cases 1000x!) that long. I even had one instance where it took 2.2442 seconds for memcached to store a value.

Obviously this is killing our site.

Here's the output of Memcached->getStats during one of the slow periods:

        [pid] => 18079
        [uptime] => 8903
        [threads] => 4
        [time] => 1332795759
        [pointer_size] => 32
        [rusage_user_seconds] => 26
        [rusage_user_microseconds] => 503872
        [rusage_system_seconds] => 125
        [rusage_system_microseconds] => 477008
        [curr_items] => 42099
        [total_items] => 422500
        [limit_maxbytes] => 943718400
        [curr_connections] => 84
        [total_connections] => 4946
        [connection_structures] => 178
        [bytes] => 7259957
        [cmd_get] => 1679091
        [cmd_set] => 351809
        [get_hits] => 1662048
        [get_misses] => 17043
        [evictions] => 0
        [bytes_read] => 109388476
        [bytes_written] => 3187646458
        [version] => 1.4.13

So things that I have ruled out so far are:

  • Hitting the max connections limit (curr_connections of 84 is well below the default of max of 1024)
  • Swapping - the machine has 900M out of 1024M of memory dedicated to memcached on a dedicated machine. It only appears to be using about 7MB of data as per the bytes stat.

How would I diagnose the other hardware problems? prstat doesn't really show a whole lot going on in terms of CPU or memory usage. Not sure how to figure out the network problems but as this is a dedicated server on the same private network as the web box I don't think it's a connectivity issue (ping is less than a millisecond between the boxes).

Is there something else I'm missing here? It's driving me nuts.

Edit: Also forgot to mention that I've tried both persistent and non-persistent connections with minimal-to-no impact.

share|improve this question
    
" I even had one instance where it took 2.2442 seconds for memcached to store a value." How big is the payload you are storing? –  Mike Purcell Mar 26 '12 at 21:19
    
Mike - It was all small test values: $to_store = json_encode(array('dummy'=>'some data','hey'=>1,'this'=>'is just fake data i am putting in to test with...')); –  Brad Dwyer Mar 26 '12 at 21:22
    
How many servers in the pool, just the one? –  Mike Purcell Mar 26 '12 at 21:37
    
Yes, just one for now. –  Brad Dwyer Mar 26 '12 at 21:40
    
Have you tried lsof (list of open files) on the memcached pid? When I end up with "lag" issues I run this command to see what exactly is going on with the process. –  Mike Purcell Mar 26 '12 at 21:54
show 5 more comments

migrated from stackoverflow.com Mar 27 '12 at 16:10

This question came from our site for professional and enthusiast programmers.

3 Answers

Memcached performance can degrade significantly if it uses swap memory. If you notice that swap memory is being used on your server, you can try starting memcached with the -k option.

From: http://code.google.com/p/memcached/wiki/NewHardware#Avoid_Swapping

Avoid Swapping

Assign physical memory, with a few percent extra, to a memcached server. Do not over-allocate memory and expect swap to save you. Performance will be very, very poor. Take extra care to monitor if your server is using swap, and tune if necessary.

share|improve this answer
    
As noted in the original post swap is not being used. There is only 7mb of data in the cache. –  Brad Dwyer Mar 31 '12 at 5:33
add comment

I changed operating systems from SmartOS to Ubuntu and the problem seems to be resolved. Not sure why but it appears to have been an issue between memcached and the OS.

share|improve this answer
add comment
up vote 0 down vote accepted

The problem ended up being that the calling machine was using up all its CPU which was causing its TCP connections to lag heavily. Scaling the web tier horizontally fixed the problem. This turned out not to be a memcached issue at all -- that's just where the symptoms of this other problem were cropping up.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.