1

I have a CentOS 7 server with Memcached. There are a lot of requests, and usually after a few hours, the memcached daemon crashes. I always restart the service, but it crashes again a few hours later.

The /var/log/memcached.log file is empty.

These are the /etc/sysconfig/memcached settings:

PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="32"
OPTIONS="-vv >> /var/log/memcached.log 2>&1"

journalctl -u memcached throws this result (strangely, the last log dates are from december, but i've been restarting the memcached service every single day in january).

Dec 10 23:59:32 myserver systemd[1]: Started Memcached.
Dec 10 23:59:32 myserver systemd[1]: Starting Memcached...
Dec 13 16:44:07 myserver systemd[1]: memcached.service: main process exited, code=killed, status=9/KILL
Dec 13 16:44:07 myserver systemd[1]: Unit memcached.service entered failed state.
Dec 13 16:44:07 myserver systemd[1]: memcached.service failed.
Dec 14 22:22:41 myserver systemd[1]: Started Memcached.
Dec 14 22:22:41 myserver systemd[1]: Starting Memcached...
Dec 15 19:35:50 myserver systemd[1]: memcached.service: main process exited, code=killed, status=9/KILL
Dec 15 19:35:50 myserver systemd[1]: Unit memcached.service entered failed state.
Dec 15 19:35:50 myserver systemd[1]: memcached.service failed.

systemctl cat memcached.service

[Unit]
Description=Memcached 
Before=httpd.service
After=network.target

[Service]
Type=simple
EnvironmentFile=-/etc/sysconfig/memcached
ExecStart=/usr/bin/memcached -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTION

[Install]
WantedBy=multi-user.target

After a few hours of visitor requests, the apache log error starts displaying this errors, one after the other:

[Wed Jan 08 11:17:11.017280 2020] [:error] [pid 15793] [client X.X.X.X:22992] PHP Warning:  Memcache::connect(): Can't connect to localhost:11211, Connection refused (111) in /www/html/inc/functions.php on line X

These are the memcached settings with a php 5.6.30 environment:

versions

How could I properly detect what's causing the fail/crash behavior?

  • 3
    I'd start by deleting >> /var/log/memcached.log 2>&1 from OPTIONS but keep -vvas you are using systemd. It should then start logging again (to journal, not to logfile). What's the status of the service in stopped state systemctl status memcached? – Lenniey Jan 8 at 17:56
  • @Lenniey after failing/crashing, the status is: Loaded: loaded (/usr/lib/systemd/system/memcached.service; enabled; vendor preset: disabled) / Active: failed (Result: signal) since Wed 2020-01-08 12:30:05 -05; 2h 51min ago / Process: 21242 ExecStart=/usr/bin/memcached -u $USER -p $PORT -m $CACHESIZE -c $MAXCONN $OPTIONS (code=killed, signal=KILL) / Main PID: 21242 (code=killed, signal=KILL) - i already removed the OPTIONS values except -vv, but after restarting memcached, the journal is still not logging anything. – Andres SK Jan 8 at 20:24
  • 1
    Can you please provide 'systemctl cat memcached.service'. Also, how are you restarting memcached? My own experience with memcached taking down a server has been with a Java client which exhausted the number of file-descriptors. – Cameron Kerr Jan 8 at 20:43
  • if you've removed the logging-to-file, and its not appearing in the journal, then check syslog, you may find it in /var/log/messages or similar. – Cameron Kerr Jan 8 at 20:45
  • 1
    You can see that its being killed with SIGKILL (kill -9 PID), so if that's not something you've been doing, something else may be doing that (eg. is it using more memory than some configuration management system is allowing?) – Cameron Kerr Jan 8 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.