3

Setup

1 redis master

3 clients redis slaves(for reads) with tunnels to master for writes

Each redis instance is a different server in a different location.

Requirement

Keep a persistent connection. When a slave disconnects be able to reconnect and resync with master as quickly as possible (< 1 minute).

sshd on master

TCPKeepAlive no

autossh configuration file

REDIS_SLAVE_PORT=6379 REDIS_MASTER_PORT=6379

AUTOSSH_FIRST_POLL=5 AUTOSSH_POLL=11

AUTOSSH_PORT=20000

AUTOSSH_GATETIME=10

AUTOSSH_LOGFILE=/home/xxx/autossh.log

AUTOSSH_PATH=/usr/bin/ssh

AUTOSSH_PIDFILE=/home/xxx/autossh.pid

AUTOSSH_LOGLEVEL=7 # FOR DEBUG

export AUTOSSH_POLL AUTOSSH_LOGFILE AUTOSSH_PATH AUTOSSH_GATETIME AUTOSSH_PORT AUTOSSH_PIDFILE AUTOSSH_FIRST_POLL AUTOSSH_LOGLEVEL

autossh -2 -fN -M ${AUTOSSH_PORT} -C -L ${REDIS_SLAVE_PORT}:localhost:${REDIS_MASTER_PORT} -i /home/xxx/.ssh/id_rsa user@master_ip

each slave has own AUTOSSH_PORT 20000, 20002 20004 etc

Problem

Usually when the connection between a slave and the master dies its re-established pretty quickly - autossh log:

autossh[pid]: timeout polling to accept read connection

autossh[pid]: port down, restarting ssh

autossh[pid]: checking for grace period, tries = 0

autossh[pid]: starting ssh (count x)

autossh[pid]: ssh child pid is xxx

autossh[pid]: check on child xxx

autossh[pid]: set alarm for 5 secs

autossh[pid]: execing /usr/bin/ssh

autossh[pid]: connection ok

However, sometimes a different kind of disconnect seems to happen which won't reconnect even after 10 minutes of trying:

autossh[pid]: 127.0.0.1:20000: Connection refused

autossh[pid]: port down, restarting ssh

Only way I found currently to make it reconnect is to change the monitor port in the autossh configuration file by hand. This is not a good solution unless I can somehow automate it and free the port.

I tried to use inetd echo service on port 7 to find a workaround for monitor port(-M 20000:7) being in use, but for some reason that didn't work here is my autossh log from the attempt:

autossh[pid]: not what I sent: "ubuntu autossh 10670 1817574984 " : ""

and after that autossh dies.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.