0

I'm having troubles with my memcached (v1.5.13) instance. It starts just fine but after a few seconds it becomes non responsive. I'm on a currently updated arch-linux box trying to access it just from localhost, so no firewall/network issues.

I tried testing with the "official" tool, this is the result:

$ systemctl start memcached.service 
$ ./mc_conn_tester.pl -s localhost -p 11211 -c 1000 --timeout 1
loop: (timeout: 1) (elapsed: 0.00663400) (conn: 0.00418401) (set: 0.00200486) (get: 0.00035810)
loop: (timeout: 1) (elapsed: 0.00163913) (conn: 0.00074816) (set: 0.00039697) (get: 0.00038695)
loop: (timeout: 1) (elapsed: 0.00193477) (conn: 0.00113797) (set: 0.00030398) (get: 0.00041795)
loop: (timeout: 1) (elapsed: 0.00166988) (conn: 0.00101185) (set: 0.00033712) (get: 0.00024605)
loop: (timeout: 1) (elapsed: 0.00124598) (conn: 0.00079703) (set: 0.00018406) (get: 0.00021100)
loop: (timeout: 1) (elapsed: 0.00123787) (conn: 0.00081491) (set: 0.00020504) (get: 0.00016499)
loop: (timeout: 1) (elapsed: 0.00171614) (conn: 0.00114703) (set: 0.00028014) (get: 0.00022793)
loop: (timeout: 1) (elapsed: 0.00115395) (conn: 0.00074410) (set: 0.00015283) (get: 0.00018811)
loop: (timeout: 1) (elapsed: 0.00132918) (conn: 0.00079608) (set: 0.00030398) (get: 0.00016999)
loop: (timeout: 1) (elapsed: 0.00126791) (conn: 0.00082994) (set: 0.00016093) (get: 0.00021601)
loop: (timeout: 1) (elapsed: 0.00120115) (conn: 0.00081515) (set: 0.00015998) (get: 0.00017190)
loop: (timeout: 1) (elapsed: 0.00153708) (conn: 0.00091696) (set: 0.00025797) (get: 0.00027800)
loop: (timeout: 1) (elapsed: 0.00208688) (conn: 0.00102091) (set: 0.00049210) (get: 0.00047183)
loop: (timeout: 1) (elapsed: 0.00124884) (conn: 0.00082898) (set: 0.00017500) (get: 0.00018001)
loop: (timeout: 1) (elapsed: 0.00228310) (conn: 0.00106001) (set: 0.00074816) (get: 0.00034499)
loop: (timeout: 1) (elapsed: 0.00214005) (conn: 0.00116301) (set: 0.00035381) (get: 0.00044703)
loop: (timeout: 1) (elapsed: 0.00207090) (conn: 0.00098491) (set: 0.00034094) (get: 0.00066209)
loop: (timeout: 1) (elapsed: 0.00130296) (conn: 0.00073791) (set: 0.00019097) (get: 0.00029707)
loop: (timeout: 1) (elapsed: 0.00242686) (conn: 0.00132298) (set: 0.00049806) (get: 0.00054789)
loop: (timeout: 1) (elapsed: 0.00280905) (conn: 0.00146294) (set: 0.00038409) (get: 0.00087500)
loop: (timeout: 1) (elapsed: 0.00151491) (conn: 0.00077105) (set: 0.00036383) (get: 0.00027919)
loop: (timeout: 1) (elapsed: 0.00125599) (conn: 0.00063396) (set: 0.00040388) (get: 0.00016403)
loop: (timeout: 1) (elapsed: 0.00157404) (conn: 0.00080013) (set: 0.00019789) (get: 0.00049305)
loop: (timeout: 1) (elapsed: 0.00114512) (conn: 0.00071216) (set: 0.00017500) (get: 0.00020289)
Fail: (timeout: 1) (elapsed: 1.00045800) (conn: 0.00885892) (set: 0.00000000) (get: 0.00000000)
Fail: (timeout: 1) (elapsed: 1.00032616) (conn: 0.00062394) (set: 0.00000000) (get: 0.00000000)
Fail: (timeout: 1) (elapsed: 1.00024891) (conn: 0.00064898) (set: 0.00000000) (get: 0.00000000)
Fail: (timeout: 1) (elapsed: 1.00029612) (conn: 0.00111318) (set: 0.00000000) (get: 0.00000000)
Fail: (timeout: 1) (elapsed: 1.00028920) (conn: 0.00076103) (set: 0.00000000) (get: 0.00000000)
Fail: (timeout: 1) (elapsed: 1.00022006) (conn: 0.00080514) (set: 0.00000000) (get: 0.00000000)
Fail: (timeout: 1) (elapsed: 1.00071883) (conn: 0.00067496) (set: 0.00000000) (get: 0.00000000)
Fail: (timeout: 1) (elapsed: 1.00066304) (conn: 0.00082493) (set: 0.00000000) (get: 0.00000000)
^CAverages: (conn: 0.03975320) (set: 0.00907159) (get: 0.00800204)

This is the -vv output on each Loop, after which it doesn't show anything during the Fails.

<28 new auto-negotiating client connection
28: Client using the ascii protocol
<28 set foo 0 0 2
>28 STORED
<28 set foo 0 0 2
>28 STORED
<28 set foo 0 0 2
>28 STORED
<28 get foo
>28 sending key foo
>28 END
<28 get foo
>28 sending key foo
>28 END
<28 get foo
>28 sending key foo
>28 END
<28 get foo
>28 sending key foo
>28 END
<28 get foo
>28 sending key foo
>28 END
<28 get foo
>28 sending key foo
>28 END
<28 connection closed.

The same is happening with memcached-tool, I can connect just fine after restarting it but only during a few seconds:

$ systemctl restart memcached.service
$ memcached-tool 127.0.0.1:11211 stats
#127.0.0.1:11211   Field       Value
         accepting_conns           1
               auth_cmds           0
             auth_errors           0
                   bytes           0
              bytes_read          14
           bytes_written        1878
              cas_badval           0
                cas_hits           0
              cas_misses           0
               cmd_flush           0
                 cmd_get           0
                 cmd_set           0
               cmd_touch           0
             conn_yields           0
   connection_structures           3
   crawler_items_checked           0
       crawler_reclaimed           0
        curr_connections           2
              curr_items           0
               decr_hits           0
             decr_misses           0
             delete_hits           0
           delete_misses           0
         direct_reclaims           0
          evicted_active           0
       evicted_unfetched           0
               evictions           0
       expired_unfetched           0
             get_expired           0
             get_flushed           0
                get_hits           0
              get_misses           0
              hash_bytes      524288
       hash_is_expanding           0
        hash_power_level          16
               incr_hits           0
             incr_misses           0
                libevent 2.1.8-stable
          limit_maxbytes    67108864
     listen_disabled_num           0
        log_watcher_sent           0
     log_watcher_skipped           0
      log_worker_dropped           0
      log_worker_written           0
       lru_bumps_dropped           0
     lru_crawler_running           0
      lru_crawler_starts         255
  lru_maintainer_juggles          52
       lrutail_reflocked           0
            malloc_fails           0
         max_connections        1024
           moves_to_cold           0
           moves_to_warm           0
        moves_within_lru           0
                     pid         921
            pointer_size          64
               reclaimed           0
    rejected_connections           0
            reserved_fds          20
           rusage_system    0.022412
             rusage_user    0.014347
   slab_global_page_pool           0
slab_reassign_busy_deletes           0
slab_reassign_busy_items           0
slab_reassign_chunk_rescues           0
slab_reassign_evictions_nomem           0
slab_reassign_inline_reclaim           0
   slab_reassign_rescues           0
   slab_reassign_running           0
             slabs_moved           0
                 threads           4
                    time  1558023648
time_in_listen_disabled_us           0
       total_connections           4
             total_items           0
              touch_hits           0
            touch_misses           0
                  uptime           5
                 version      1.5.13

Then when I retry it just hangs up there waiting:

$ memcached-tool 127.0.0.1:11211 stats

I'm totally lost.

1

Had the same issue. Logging didn't show anything useful.

In my case the issue was too many open client connections were (left) open so new ones couldn't be established. On the client side this usually does not get shown or logged (though it depends on the client library).

memcached-tool stops working because at the end of the day it's also just another TCP connection, trying to fetch stats.

You can check the current list of clients with netstat using something like:

netstat | grep memcached | grep ESTAB | wc -l

See if this is too high (default is max 1024 or so?) and/or who the clients are.

  • Sadly clients are just a few local applications so I don't think thats my case, but I'll check open file descriptors and stuff like that that I hadn't thought about before your post. – Mauricio Pasquier Juan Jun 5 '19 at 21:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.