PHP - Memcached - Libmemcached - Handle cache server outage

Question

I am working on ensuring our app degrades gracefully in case of a complete cache outage, which is highly unlikely, as we have minimum of at least 3 cache nodes to add to the cache pool, by way of PHP's memcached addServer api call. However, it is possible that a single node may fail, and I need to ensure that the memcached api correctly handles this scenario.

Here is my current cache.yml config

port: 11211
<?php echo Hobis_Api_Cache::TYPE_VOLATILE; ?>:
  options:
    - <?php echo Memcached::OPT_CONNECT_TIMEOUT; ?>: 25<?php echo PHP_EOL; ?>
    #- <?php echo Memcached::OPT_DISTRIBUTION; ?>: <?php echo Memcached::DISTRIBUTION_CONSISTENT; ?><?php echo PHP_EOL; ?>
    - <?php echo Memcached::OPT_LIBKETAMA_COMPATIBLE; ?>: true<?php echo PHP_EOL; ?>
    - <?php echo Memcached::OPT_NO_BLOCK; ?>: true<?php echo PHP_EOL; ?>
    #- <?php echo Memcached::OPT_POLL_TIMEOUT; ?>: 100<?php echo PHP_EOL; ?>
    #- <?php echo Memcached::OPT_RECV_TIMEOUT; ?>: 10000<?php echo PHP_EOL; ?>
    - <?php echo Memcached::OPT_REMOVE_FAILED_SERVERS; ?>: true<?php echo PHP_EOL; ?>
    - <?php echo Memcached::OPT_RETRY_TIMEOUT; ?>: 1<?php echo PHP_EOL; ?>
    #- <?php echo Memcached::OPT_SEND_TIMEOUT; ?>: 10000<?php echo PHP_EOL; ?>
    - <?php echo Memcached::OPT_SERIALIZER; ?>: <?php echo Memcached::SERIALIZER_IGBINARY; ?><?php echo PHP_EOL; ?>
    #- <?php echo Memcached::OPT_SERVER_FAILURE_LIMIT; ?>: 1<?php echo PHP_EOL; ?>
    - <?php echo Memcached::OPT_TCP_NODELAY; ?>: true<?php echo PHP_EOL; ?>
  servers:
    - vcache-1
    - vcache-2
    - vcache-3
<?php echo Hobis_Api_Cache::TYPE_PERSISTENT; ?>:
  servers:
    - pcache-1

Based on some research (here and here), it's possible for the memcached api to gracefully handle a single node outage, if part of a pool. However in my case, I am unable to write specific keys during testing. Instead I get an 'unable to write' error with a resultCode of 35, which according to the comments, is MEMCACHED_SERVER_MARKED_DEAD.

Having a server marked dead is indeed what I would expect, as I stopped vcache-2/3 and only vcache-1 is running, however, with the OPT_LIBKETAMA_COMPATIBLE option, I am under the impression that the memcached api is to write the key to another server in the pool. And that with the OPT_REMOVE_FAILED_SERVERS option, I shouldn't be seeing the marked dead result codes, because the server should be removed from the pool.

Any suggestions?

Answer 1

Finally, I have a working solution in place, and sharing with others in case they encounter the same sitch. The basis for my solution came from this post, as soon as I saw the $testInstance and $realInstance it dawned on me, use the test instance to determine which servers are available, then add known good servers to the real instance. You might be asking yourself, why not just call addServers twice against the same instance, the answer? You can't.

If you try calling addServers more than once against the same instance you should see unexpected behavior, for me it was a bad gateway due to: " upstream prematurely closed connection while reading response header from upstream". So there was some internal mechanism causing PHP to fail out, although I don't know why specifically, as no errors showed up in the error log.

Rather than make explicit connect calls as the author did, I opted to use the available getStats() call, and check for a pid.

Working snippet:

$cacheReal = new Memcached;
$cacheTest = new Memcached;

if (count($cacheTest->getServerList()) < 1) {

    $knownGoodServers   = array();
    $serversToAdd       = array();

    foreach ($servers as $server) {             
        $serversToAdd[] = array($server, $port);
    }

    $cacheTest->addServers($serversToAdd);

    foreach ($cacheTest->getStats() as $server => $stats) {

        // Test if server is actually available
        if ((false === Hobis_Api_Array_Package::populatedKey('pid', $stats)) ||
            ($stats['pid'] < 0)) {
            continue;
        }

        $knownGoodServers[] = $server;
    }

    // It is possible that entire cache pool took a dump
    if (true === Hobis_Api_Array_Package::populated($knownGoodServers)) {

        $serversToAdd = array();

        foreach ($knownGoodServers as $server) {

            list($host, $port) = array_map('trim', explode(':', $server));

            $serversToAdd[] = array($host, (int) $port);
        }

        if (true === Hobis_Api_Array_Package::populated($serversToAdd)) {
            $cacheReal->addServers($serversToAdd);
        }
    }
}

---

In my testing I had multiple cache nodes which were all running for test 1, then for subsequent tests, I switched some cache nodes off and on, then finally all off (to test degradation). The only noticeable difference was that after taking a node down, or adding a node back (via daemon restart), my logged in session would be logged out, which makes sense, as the cached data is no longer available on expected server. However subsequent requests after logging back in demonstrated expected behavior while logged in, because the session data was written to cache nodes available at the time of the request.