Memcache service is down - how to handle?

Question

I am using Memcache in my ASP.NET application. I have 3 memcache servers that it connects to. If all of them are down, there is a significant delay in loading the pages. When all servers are down, I would want to not use memcache.

When I was stepping through the code, I noticed that when the servers are down, memcache code does not throw an exception (I thought I could "Catch" it in my application and take the necessary steps).

How would I handle this situation?

EDIT: More Details. My config file looks like this:

We have changed the default setting of deadTimeOut from 2 mins to 10 sec so that dead servers are resurrected asap (What is the optimum setting?). I thought of way of implementing this and wanted to run it by you all. Basically, here's the requirement:

1. If a server goes down - alert the admin
2. Turn off the timer (isAliveTimer) that checks whether the server is up or not every 10 seconds
3. Do not use memcache if ANY of the servers are down. My understanding is that if out of 2, 1 is down, we still have an issue with the timer checking whether the dead server is up or not which leads to bad response time.

I currently use the following code in my application:

private class Singleton
{
  private static MemcachedClient instance;

  protected Singleton()
  {
  }

  public static MemcachedClient Instance()
  {
    if (instance == null)
    {
      instance = new MemcachedClient();
    }

    return instance;
  }
}

To create an instance, I do:

MemcachedClient mc = Singleton.Instance();

I can look at mc.pool.WorkingServers.Items[0].IsAlive to check if the first server is alive or not. If it's false, I can send the alert (Takes care of requirement 1). However, what if the server goes down after getting the instance of MemcachedClient? Since I use a singleton object, I don't think I will be able to take care of the use case in which the server would go down after creating the instance.

I am not sure what to do about 2 and 3.

It seems to me that our requirement is fairly simple. Since memcache is used by companies with larger loads, I wonder how they take care of the scenario of sending an admin alert and preventing the TCP timeouts.

Any help is appreciated! Thank you

Answer 1

Was basically going to recommend what Chris Kinsman just posted. I'd add that you can also use the separate thread to maintain a pool of active connections to your memcache servers, so you're not using up resources constantly creating and destroying connections. This tends to have advantages under heavy load, as well as generally being a little more elegant.

Answer 2

There is nothing in memcache to see connection issue since you are dealing with tcp timeouts. The best thing to do is have machine be able to do IP stealing from one another in case there is failure of a single device.

Answer 3

Since you don't want to wait for the TCP timeout to occur could you spin up a separate thread on application start that is checking the health of hte memcached instances? If the instance isn't responding it sets a global flag in the instance that your code can check before attempting to call memcached?

This allows you do the checks in an async fashion and throttle up and down based on instance availability.