Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Question

I am interested in learning the following metrics, per memcached instance:

  • Safe range for connections/GETs/SETs per second
  • Upper bound for connections/GETs/SETs per second

I have a feeling the true bottleneck will be connections, but would like input from people who have established installations of memcached on their sites.

Background

I run a website that is seeing hundreds of millions of page views over the course of a month. It is distributed across multiple web servers. The site was originally coded with a file-based cached schema that was not shared by the web server pool; each web server maintained its own cached copy of each page.

For obvious reasons, we're migrating to memcached. We converted our lower-traffic but more-dynamic pages (aka 'pages with lower cache hit rates') without issue. We're now moving on to our higher-traffic but more-static pages (aka 'pages that should see higher cache hit rates'). We've converted the lowest-traffic ones first and we've already seen a jump from 3.5k GETs per second on average to 11k GETs per second on average. We're seeing between 400-600 connections active at any given time on average. Our connection limit is set to 4k in our configuration file.

Considering we still have the highest-traffic pages left to implement, this seemed like a good time to research the accepted ranges and upper limits regarding memcached. That way, we can determine if we need to expand to additional memcached instances before we move the highest-traffic portions of our site over to memcached. I realize our usage now is no cause for alarm, but I'd like to know when it will be, and I'd like to know that beforehand.

share|improve this question
    
Are you caching the whole pages or just SQL queries/other smaller objects? –  Alex Feb 7 '11 at 23:50
    
@Alex: A combination of objects (where we see them shared in many places across pages) and queries (for specific filtered lists or other areas where the cached value wouldn't be shared across pages). We're not trying to serialize pages and store them in memcached. :) –  Shaun Feb 7 '11 at 23:52

3 Answers 3

up vote 1 down vote accepted

A number of concurrent connections is more important than a number of GETs/SETs from my experience. I am looking at a historical Cacti graph right now, the graph states that memcached instance got about 4M accesses per second at max (2.8M GETs and 1.2M SETs). I doubt these numbers are real. They were achieved using only one active connection anyway. The problem was that when I stress tested this setup using a web site load testing tool, memcached started to hog CPU at only about 2-3K accesses per second. I had to build a farm of memcacheds to distribute the load. As you can see, there is a non-linear relationship between number of concurrent connections and number of accesses per second memcached can handle. Memcached seems to start degrading quickly on a certain number of concurrent connections, you really need to stress test your setup to determine this critical number.

share|improve this answer

Perhaps a better way to look at it is to see your limitation on performance interacting with memcached. Some scripts to ascertain the average response time across your typical day would be a good place to start.

On your main question:

I think in general the amount of concurrent activities (gets/sets) will vary on your hardware configuration anyway so any responses on that would be subjective. I think finding a better way to measure the success relative to your own environment to get right to the point. As noted, stress testing is naturally very important in that process.

Cheers

share|improve this answer
    
Stress testing is of course a valid point, and you get a +1 for that. In addition to that, however, I am looking for what a typical installation sees in terms of bottlenecks. –  Shaun Feb 14 '11 at 19:16

There's no magic formula for predicting the capacity - you need to do this by experimentation and numeric analysis.

we've already seen a jump from 3.5k GETs per second on average to 11k GETs per second on average

If you're saying that this is the result of implementing the change, that implies that you were previously losing 2 thirds of your traffic due to performance issues?

While these metrics suggest you may be able to service more traffic with less hardware, did you measure the effect on response times? Using appropriate network simulation?

You need to plan an architecture which allows you to control what and where the caching is done - you want to be able to easily distribute the load across multiple memcached instances with some capability for failover. This gets very complicated, very quickly - which is one of the reasons I tend to leave memcached till a last resort - and never for a front-end cache. The other reasons is that where I've tested it, there is negligible gains compared with other methods of sharing data (reverse proxies at the front, good tuning in the middle, and distributed, shared storage at the back end). But I'm very conscious that the architecture needs to reflect the components used to build the system - my experience is mostly with mid-scale LAMP type systems - but model with a split application server tier is very different.

share|improve this answer
1  
He means memcache GETs, not HTTP GETs –  Andreas Bonini Feb 8 '11 at 12:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.