7

We rely heavily on memcache and are serving a few billion requests per month. We have 5 memcache servers. Last night, we saw an 25% increase in our traffic. The graphs show that requests and data transfered by each memcache increased and made them crash. It started a chain reaction and each memcache server crashed one after another (Load per server increased).

We found no logs in syslog, messages, memcache log file (Verbose settings was off).

I have two questions:

  1. How can I find out why exactly this happened. If load is an issue for memcache, is there any documentation on how much a normal memcache (running on decent config) can handle. How can I increase this value.

  2. How can I ensure they never go down again. It eventually impacted our mysql servers and replication and impacted a lot of other related services. Do I need more memcache servers?

I started my memcache using this init.d script: http://pastebin.com/wfMnB4ta where ENABLE_MEMCACHE is YES in /etc/default/memcached

/usr/share/memcached/scripts/start-memcached: http://pastebin.com/LaUugXye

Thanks

Improve this question
12
  • How did you start memcached?
    – quanta
    Oct 27, 2011 at 6:12
  • I used the default init.d scripts to start memcache
    – Sparsh Gupta
    Oct 27, 2011 at 6:17
  • Append your init script to the first post.
    – quanta
    Oct 27, 2011 at 6:21
  • Done, its pastebin.com/wfMnB4ta
    – Sparsh Gupta
    Oct 27, 2011 at 6:32
  • What does /usr/share/memcached/scripts/start-memcached look like?
    – quanta
    Oct 27, 2011 at 6:40

2 Answers 2

Reset to default
6

I'm going to guess that you run version 1.4.5 or older.

Since you mention an increase in traffic, then a sudden exit:

  • You may have hit the max connections limit (see http://memcached.org/timeouts for some discussion on this).
  • If you hammer the connection limit for a long time, there was a bug which would cause memcached to exit.
  • This was partially repaired in 1.4.6, further repaired in 1.4.7, and refined through 1.4.9.

If you ever experience a crash, the first thing to do is make sure you're on the latest stable release. If you still experience crashes, the best thing to do is to contact the actual mailing list or file a bug report with information, rather than get lucky with a maintainer seeing this via a twitter search.

Doing periodic upgrades to match the latest stable can help you avoid having your whole cluster crash in the future.

Improve this answer
2
  • Thanks, I was on memcache 1.4.2, I will upgrade it first thing now
    – Sparsh Gupta
    Oct 27, 2011 at 6:37
  • 2
    i'm facing the same problem and i'm using 1.4.15
    – khizar ansari
    Mar 30, 2013 at 9:55
0

You should also work out some kind of structural solution to deal with similar problems. For example, if you notice that the response time on requests is increasing, reduce the number of requests. You can do this various ways, including disabling non-essential services.

This particular failure would likely not have been avoidable though. There's not much you can about a failure causing increasing load.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .