Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We rely heavily on memcache and are serving a few billion requests per month. We have 5 memcache servers. Last night, we saw an 25% increase in our traffic. The graphs show that requests and data transfered by each memcache increased and made them crash. It started a chain reaction and each memcache server crashed one after another (Load per server increased).

We found no logs in syslog, messages, memcache log file (Verbose settings was off).

I have two questions:

  1. How can I find out why exactly this happened. If load is an issue for memcache, is there any documentation on how much a normal memcache (running on decent config) can handle. How can I increase this value.

  2. How can I ensure they never go down again. It eventually impacted our mysql servers and replication and impacted a lot of other related services. Do I need more memcache servers?

I started my memcache using this init.d script: http://pastebin.com/wfMnB4ta where ENABLE_MEMCACHE is YES in /etc/default/memcached

/usr/share/memcached/scripts/start-memcached: http://pastebin.com/LaUugXye

Thanks

share|improve this question
    
How did you start memcached? –  quanta Oct 27 '11 at 6:12
    
I used the default init.d scripts to start memcache –  Sparsh Gupta Oct 27 '11 at 6:17
    
Append your init script to the first post. –  quanta Oct 27 '11 at 6:21
    
Done, its pastebin.com/wfMnB4ta –  Sparsh Gupta Oct 27 '11 at 6:32
    
What does /usr/share/memcached/scripts/start-memcached look like? –  quanta Oct 27 '11 at 6:40
show 7 more comments

2 Answers

up vote 5 down vote accepted

I'm going to guess that you run version 1.4.5 or older.

Since you mention an increase in traffic, then a sudden exit:

  • You may have hit the max connections limit (see http://memcached.org/timeouts for some discussion on this).
  • If you hammer the connection limit for a long time, there was a bug which would cause memcached to exit.
  • This was partially repaired in 1.4.6, further repaired in 1.4.7, and refined through 1.4.9.

If you ever experience a crash, the first thing to do is make sure you're on the latest stable release. If you still experience crashes, the best thing to do is to contact the actual mailing list or file a bug report with information, rather than get lucky with a maintainer seeing this via a twitter search.

Doing periodic upgrades to match the latest stable can help you avoid having your whole cluster crash in the future.

share|improve this answer
    
Thanks, I was on memcache 1.4.2, I will upgrade it first thing now –  Sparsh Gupta Oct 27 '11 at 6:37
    
i'm facing the same problem and i'm using 1.4.15 –  khizar ansari Mar 30 '13 at 9:55
add comment

You should also work out some kind of structural solution to deal with similar problems. For example, if you notice that the response time on requests is increasing, reduce the number of requests. You can do this various ways, including disabling non-essential services.

This particular failure would likely not have been avoidable though. There's not much you can about a failure causing increasing load.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.