2

I have a 2 node Redis cluster set-up.

[master] 192.168.56.102: Redis Master (:6379), Redis Slave (:6380), Sentinel(:26379), Sentinel#2(:26380)

[rescue] 192.168.56.103: Redis Master (:6379), Redis Slave (:6380), Sentinel(:26379)

Each slave instance is a slave of the master instance on the same machine. Each sentinel instance monitors both master instances.

I am using the above in conjunction with twemproxy (which has nothing to do with this question) and a client-reconfig-script to update twemproxy configuration so that the application keeps working.

I am stopping servers to see what is happening and if everything is working correctly.

[master] stop redis master: The quorum is able to elect a new master successfully. Log below.

==> /tmp/sentinel.log <==
[14701] 29 Dec 18:16:55.096 # +sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:16:55.096 # +sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14701] 29 Dec 18:18:04.187 # -sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:18:04.236 # -sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:18:14.160 * +convert-to-slave slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:47:47.151 # +sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14701] 29 Dec 18:47:47.170 # +sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14701] 29 Dec 18:47:57.650 * +reboot slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:47:57.652 * +reboot slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:47:57.715 # -sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14701] 29 Dec 18:47:57.738 # -sdown slave 192.168.56.102:6379 192.168.56.102 6379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:48:08.088 # +sdown master master 192.168.56.102 6380
[14701] 29 Dec 18:48:08.180 # +sdown master master 192.168.56.102 6380
[14701] 29 Dec 18:48:08.280 # +odown master master 192.168.56.102 6380 #quorum 2/2
[14701] 29 Dec 18:48:08.280 # +new-epoch 73
[14701] 29 Dec 18:48:08.280 # +try-failover master master 192.168.56.102 6380
[14701] 29 Dec 18:48:08.471 # +vote-for-leader a664b9f61df2b10bbbb5d865b01c599ddd36183c 73
[14701] 29 Dec 18:48:08.472 # 192.168.56.103:26379 voted for 491b32b95c547a8266faf9b04ce6b0c18486236b 73
[14705] 29 Dec 18:48:08.473 # +new-epoch 73
[14705] 29 Dec 18:48:08.475 # +vote-for-leader 491b32b95c547a8266faf9b04ce6b0c18486236b 73
[14701] 29 Dec 18:48:08.475 # 192.168.56.102:26380 voted for 491b32b95c547a8266faf9b04ce6b0c18486236b 73
[14701] 29 Dec 18:48:08.835 # +config-update-from sentinel 192.168.56.103:26379 192.168.56.103 26379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:48:08.835 # +config-update-from sentinel 192.168.56.103:26379 192.168.56.103 26379 @ master 192.168.56.102 6380
[14705] 29 Dec 18:48:08.836 # +switch-master master 192.168.56.102 6380 192.168.56.102 6379
[14701] 29 Dec 18:48:08.836 # +switch-master master 192.168.56.102 6380 192.168.56.102 6379
[14701] 29 Dec 18:48:08.836 * +slave slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379
[14705] 29 Dec 18:48:08.836 * +slave slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379

==> /tmp/sent.log <==
Failing master: 192.168.56.102:6380
New Master: 192.168.56.102:6379

Failing master: 192.168.56.102:6380
New Master: 192.168.56.102:6379


==> /tmp/sentinel.log <==
[14705] 29 Dec 18:48:11.855 # +sdown slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379
[14701] 29 Dec 18:48:11.903 # +sdown slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379

You can also see that my reconfigure script is called successfully and outputs some details on the screen.

The problem is when i am trying to stop the master instance on the [rescue] machine. The sentinel continuously reports "-failover-abort-not-elected master resque 192.168.56.103 6379"

==> /tmp/sentinel.log <==
[14705] 29 Dec 18:48:11.855 # +sdown slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379
[14701] 29 Dec 18:48:11.903 # +sdown slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379
[14705] 29 Dec 18:48:43.401 # -sdown slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379
[14701] 29 Dec 18:48:43.433 # -sdown slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379
[14705] 29 Dec 18:48:53.344 * +convert-to-slave slave 192.168.56.102:6380 192.168.56.102 6380 @ master 192.168.56.102 6379
[14705] 29 Dec 18:49:23.617 # +sdown master resque 192.168.56.103 6379
[14701] 29 Dec 18:49:23.625 # +sdown master resque 192.168.56.103 6379
[14705] 29 Dec 18:49:23.674 # +odown master resque 192.168.56.103 6379 #quorum 2/2
[14705] 29 Dec 18:49:23.674 # +new-epoch 74
[14705] 29 Dec 18:49:23.674 # +try-failover master resque 192.168.56.103 6379
[14701] 29 Dec 18:49:23.727 # +odown master resque 192.168.56.103 6379 #quorum 3/2
[14701] 29 Dec 18:49:23.727 # +new-epoch 74
[14701] 29 Dec 18:49:23.727 # +try-failover master resque 192.168.56.103 6379
[14705] 29 Dec 18:49:23.886 # +vote-for-leader b608fcab7a201799826f4d9ee839aed3cf556fdf 74
[14701] 29 Dec 18:49:23.889 # +vote-for-leader a664b9f61df2b10bbbb5d865b01c599ddd36183c 74
[14701] 29 Dec 18:49:23.890 # 192.168.56.102:26380 voted for b608fcab7a201799826f4d9ee839aed3cf556fdf 74
[14705] 29 Dec 18:49:23.890 # 192.168.56.102:26379 voted for a664b9f61df2b10bbbb5d865b01c599ddd36183c 74
[14705] 29 Dec 18:49:23.893 # 192.168.56.103:26379 voted for b608fcab7a201799826f4d9ee839aed3cf556fdf 74
[14701] 29 Dec 18:49:23.893 # 192.168.56.103:26379 voted for b608fcab7a201799826f4d9ee839aed3cf556fdf 74
[14705] 29 Dec 18:49:23.980 # +elected-leader master resque 192.168.56.103 6379
[14705] 29 Dec 18:49:23.980 # +failover-state-select-slave master resque 192.168.56.103 6379
[14705] 29 Dec 18:49:24.064 # -failover-abort-no-good-slave master resque 192.168.56.103 6379
[14705] 29 Dec 18:49:24.117 # Next failover delay: I will not start a failover before Mon Dec 29 18:49:32 2014
[14701] 29 Dec 18:49:28.417 # -failover-abort-not-elected master resque 192.168.56.103 6379
[14701] 29 Dec 18:49:28.489 # Next failover delay: I will not start a failover before Mon Dec 29 18:49:32 2014
[14705] 29 Dec 18:49:32.217 # +new-epoch 75
[14705] 29 Dec 18:49:32.217 # +try-failover master resque 192.168.56.103 6379
[14701] 29 Dec 18:49:32.423 # +new-epoch 75
[14701] 29 Dec 18:49:32.424 # +try-failover master resque 192.168.56.103 6379
[14705] 29 Dec 18:49:32.433 # +vote-for-leader b608fcab7a201799826f4d9ee839aed3cf556fdf 75
[14705] 29 Dec 18:49:32.435 # 192.168.56.103:26379 voted for 491b32b95c547a8266faf9b04ce6b0c18486236b 75
[14701] 29 Dec 18:49:32.437 # +vote-for-leader a664b9f61df2b10bbbb5d865b01c599ddd36183c 75
[14701] 29 Dec 18:49:32.438 # 192.168.56.102:26380 voted for b608fcab7a201799826f4d9ee839aed3cf556fdf 75
[14705] 29 Dec 18:49:32.438 # 192.168.56.102:26379 voted for a664b9f61df2b10bbbb5d865b01c599ddd36183c 75
[14701] 29 Dec 18:49:32.438 # 192.168.56.103:26379 voted for 491b32b95c547a8266faf9b04ce6b0c18486236b 75
[14705] 29 Dec 18:49:36.636 # -failover-abort-not-elected master resque 192.168.56.103 6379
[14705] 29 Dec 18:49:36.691 # Next failover delay: I will not start a failover before Mon Dec 29 18:49:40 2014
[14701] 29 Dec 18:49:36.843 # -failover-abort-not-elected master resque 192.168.56.103 6379
[14701] 29 Dec 18:49:36.905 # Next failover delay: I will not start a failover before Mon Dec 29 18:49:40 2014

A new master is not elected and the reconfigure script is not called.

I understand that for a new master to be elected, the quorum (n/2+1) will have to agree. This is why I am using 3 sentinels to test.

I don't understand why the above example does not end up in an election (like [master]).

I am using Redis server v=2.8.19 sha=0a21368c:1 malloc=jemalloc-3.6.0 bits=64 build=e570b291804f6e35

Thanks for any help and please don't mind the gross misspelling of the word rescue!

--edit

[master] redis slave config:

daemonize yes
pidfile "/var/run/redis/redis-server-slave.pid"
port 6380
tcp-backlog 511
bind 192.168.56.102
timeout 0
tcp-keepalive 0
loglevel notice
logfile "/var/log/redis/redis-server.log"
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename "dump.rdb"
dir "/var/lib/redis"
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes
maxclients 4064
# Generated by CONFIG REWRITE

slaveof 192.168.56.102 6379

[master] master instance run_id: 5c1ffb7742ad78cde12dbe4858747a314adaebe9

[master] slave instance run_id: 5772f437519bb38782d38f6675ae5d9157be2419

[rescue] master instance run_id: 5c1ffb7742ad78cde12dbe4858747a314adaebe9

[rescue] slave instance run_id: 5772f437519bb38782d38f6675ae5d9157be2419

--

[master] Sentinel (:26379) run_id: a664b9f61df2b10bbbb5d865b01c599ddd36183c

[master] Sentinel (:26380) run_id: b608fcab7a201799826f4d9ee839aed3cf556fdf

[slave] Sentinel (:26379) run_id: a664b9f61df2b10bbbb5d865b01c599ddd36183c

Sentinel information

# Sentinel
sentinel_masters:2
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
master0:name=resque,status=ok,address=192.168.56.103:6379,slaves=0,sentinels=3
master1:name=master,status=ok,address=192.168.56.102:6379,slaves=1,sentinels=3

Sentinel conf of 192.168.56.103

port 26379
logfile "/tmp/sentinel.log"
dir "/tmp"

sentinel monitor resque 192.168.56.103 6379 2
sentinel down-after-milliseconds resque 3000
sentinel failover-timeout resque 4000
sentinel client-reconfig-script resque /home/sm0ke/Projects/git/thrace/extra/sentinel-failover.py

sentinel config-epoch resque 0
sentinel leader-epoch resque 76
sentinel known-sentinel resque 192.168.56.102 26379 a664b9f61df2b10bbbb5d865b01c599ddd36183c
sentinel known-sentinel resque 192.168.56.102 26380 b608fcab7a201799826f4d9ee839aed3cf556fdf

maxclients 4064
daemonize yes
# Generated by CONFIG REWRITE
sentinel monitor master 192.168.56.102 6379 2
sentinel down-after-milliseconds master 3000
sentinel failover-timeout master 4000
sentinel client-reconfig-script master /home/sm0ke/Projects/git/thrace/extra/sentinel-failover.py
sentinel config-epoch master 73
sentinel leader-epoch master 73
sentinel known-slave master 192.168.56.102 6380
sentinel known-sentinel master 192.168.56.102 26380 b608fcab7a201799826f4d9ee839aed3cf556fdf
sentinel known-sentinel master 192.168.56.102 26379 a664b9f61df2b10bbbb5d865b01c599ddd36183c
sentinel current-epoch 76

redis-cli info for Sentinel on 192.168.56.103

# Server
redis_version:2.8.19
redis_git_sha1:0a21368c
redis_git_dirty:1
redis_build_id:e570b291804f6e35
redis_mode:sentinel
os:Linux 3.2.0-4-amd64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.7.2
process_id:8345
run_id:491b32b95c547a8266faf9b04ce6b0c18486236b
tcp_port:26379
uptime_in_seconds:8054
uptime_in_days:0
hz:17
lru_clock:10594345
config_file:/etc/redis/sentinel.conf

# Sentinel
sentinel_masters:2
sentinel_tilt:0
sentinel_running_scripts:16
sentinel_scripts_queue_length:4
master0:name=resque,status=ok,address=192.168.56.103:6379,slaves=0,sentinels=3
master1:name=master,status=ok,address=192.168.56.102:6379,slaves=1,sentinels=3
  • What's the config look like for the 6380 instance - particularly, is there a slave-priority set? Also, what are all of these instances - there are votes for 491b32b, a664b9, and b608fc - check a sentinel's config file for the resque instance for what listeners those correspond to, something's funky if there are votes for 3 different instances. The failover-abort-no-good-slave isn't concerning, that's expected when the majority voted for the instance that just failed - the question is why those sentinels think they should be voting for it. – Shane Madden Dec 29 '14 at 18:21
  • Both slave instances have a slave-priority of 100. I have updated my question with the [master] redis slave configuration. – sm0ke21 Dec 29 '14 at 18:36
  • Is that the slave on the master node or the slave on the rescue node? And can you check what those instances IDs are? – Shane Madden Dec 29 '14 at 18:38
  • It is the slave of the master node. As far as instance IDs are concerned, i don't know what those are. I am googling it right now. – sm0ke21 Dec 29 '14 at 18:42
  • I have updated my question with some more information (run_ids). These seem to be the same for both masters and both slaves recpectively. Is this a problem? I could not find out what instance IDs are Shane. – sm0ke21 Dec 29 '14 at 18:47
3

Ignore me on the instance ID confusion, that's my fault - I forgot that the sentinels were voting for a leader among the sentinels, not voting for a new master among the candidates, so the three different IDs makes sense.

So, here's the real problem:

master0:name=resque,status=ok,address=192.168.56.103:6379,slaves=0,sentinels=3

There's no slaves there that the sentinels have noticed, so they don't have a good node to make live during the vote.

Check the slaveof config on the 192.168.56.103:6380 instance, make sure that instance is running, and connect to it to check its info to verify that it's slaving. Once the 192.168.56.103:6379 sees it as a slave (in its info command), the sentinels will pick it up as a known slave and be able to fail over to it.

  • I saw that too. It seems (for some reason that is beyond me) that the [rescue] slave was a slave of the [master] master redis instance. I have corrected that by issuing a slaveof 192.168.56.103 6379 and now it seems that the failover is done successfully. – sm0ke21 Dec 29 '14 at 19:46
  • @sm0ke21 Great! – Shane Madden Dec 29 '14 at 19:47
  • 4
    Hello. Im having the same problem but in my case I have master0:name=redis-master,status=ok,address=172.29.245.6:6379,slaves=2,sentinels=3 but Im still unable to failover. I still have +vote-for-leader 61c51a37f2db1673ddcf5dc3fe6816c9ba83a408 42 -failover-abort-not-elected master redis-master 172.29.245.6 6379 – Jason Stanley Oct 27 '17 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.