0

We have a problem with Redis in our server.

first in PHP script we connected to Redis via tcp:127.0.0.1:6379 port for caching. in this case and with netstat -anp |grep 6379 |wc -l command we had about 57000 connection and PHP couldn't connect to Redis.

after that we changed tcp to unixsocket connection. everything looks OK until we found Redis has a lot of ideal client connections.

bellow you can see output of client list command on redis:

    id=4369214 addr=/tmp/redis.sock:0 fd=953 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369215 addr=/tmp/redis.sock:0 fd=955 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369216 addr=/tmp/redis.sock:0 fd=1045 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369217 addr=/tmp/redis.sock:0 fd=1056 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369218 addr=/tmp/redis.sock:0 fd=1108 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369219 addr=/tmp/redis.sock:0 fd=1109 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369220 addr=/tmp/redis.sock:0 fd=1110 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369221 addr=/tmp/redis.sock:0 fd=1112 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369222 addr=/tmp/redis.sock:0 fd=1113 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369223 addr=/tmp/redis.sock:0 fd=1114 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369224 addr=/tmp/redis.sock:0 fd=1115 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369225 addr=/tmp/redis.sock:0 fd=1116 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369226 addr=/tmp/redis.sock:0 fd=1117 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369227 addr=/tmp/redis.sock:0 fd=1118 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369228 addr=/tmp/redis.sock:0 fd=1119 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369229 addr=/tmp/redis.sock:0 fd=1120 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369230 addr=/tmp/redis.sock:0 fd=1121 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369231 addr=/tmp/redis.sock:0 fd=1122 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369232 addr=/tmp/redis.sock:0 fd=1123 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369233 addr=/tmp/redis.sock:0 fd=1124 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369234 addr=/tmp/redis.sock:0 fd=1125 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369235 addr=/tmp/redis.sock:0 fd=1126 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369236 addr=/tmp/redis.sock:0 fd=1127 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369237 addr=/tmp/redis.sock:0 fd=1128 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369238 addr=/tmp/redis.sock:0 fd=1129 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369239 addr=/tmp/redis.sock:0 fd=1130 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369240 addr=/tmp/redis.sock:0 fd=1131 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369241 addr=/tmp/redis.sock:0 fd=1132 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369242 addr=/tmp/redis.sock:0 fd=1133 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369243 addr=/tmp/redis.sock:0 fd=1134 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369244 addr=/tmp/redis.sock:0 fd=1135 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
    id=4369245 addr=/tmp/redis.sock:0 fd=1136 name= age=0 idle=0 flags=U db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
...

and the output of info clients would be :

# Clients
connected_clients:917
client_longest_output_list:2
client_biggest_input_buf:0
blocked_clients:0

Our cup usage is high and Redis is slow. even after we disable caching we still have this issue.

thanks

0

Redis has timeout in its config. Dynamically set timeout using

 config set timeout <valid value in seconds> 

Redis closes connection if it's idle for configured time. This issue happens because your PHP script is not closing redis sockets at the end of request. I think redis slows down as it is hitting open file descriptor limit

4
  • thanks for answer. I've set it to 30 seconds but it doesn't affect anything – Saeed M. Jan 15 '17 at 13:59
  • What happens when you restart PHP process fpm or Apache? – kalyan Jan 15 '17 at 14:23
  • The redis connection closed but after about 1 hour all of them get back – Saeed M. Jan 15 '17 at 18:23
  • So apps are using so many connections. Because if they were idle for 30s or more redis would have reaped them. Could you check why so many connections opened from apps and how are they kept alive. Tcpdump can help if TCP connections are used. Strace can help for domain sockets – kalyan Jan 15 '17 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.