2

This may be a Serverfault or a Stack Overflow question, I'm not sure yet:

I've set up a simple three node Redis system with a master and two replica nodes, managing failover with Redis Sentinel. Redis and Sentinel network traffic is secured using Redis built-in TLS support and regular CA issued certificates.

Each Sentinel instance is configured to announce its hostname and to resolve DNS:

sentinel resolve-hostnames yes
sentinel announce-hostnames yes
sentinel announce-ip "redistest2.mydomain.com"

We have a web service using Servicestack to connect to the Sentinel instances. As long as we don't validate TLS certificates and hostnames everything works well: The web service can see the Redis Sentinel listeners, and when we terminate the current Master, the Redis cluster votes on a new one, and the web service switches to the new writable Redis node.

However, while the original Master node reports in with its FQDN, the two backup nodes only ever seem to report back to ServiceStack with their IP addresses.

A Sentinel log excerpt indicates that the backup nodes seem to use their host names:

28011:X 15 Feb 2023 15:23:10.817 * +sentinel sentinel <hex-string> redistest2.mydomain.com 26379 @ redistest redistest1.mydomain.com 6379
28011:X 15 Feb 2023 15:23:10.821 * Sentinel new configuration saved on disk
28011:X 15 Feb 2023 15:23:10.897 * +sentinel sentinel <other-hex-string> redistest3.mydomain.com 26379 @ redistest redistest1.mydomain.com 6379
28011:X 15 Feb 2023 15:23:10.901 * Sentinel new configuration saved on disk

Yet ServiceStack insists that it only receives the server IP addresses back from the server group:

Starting with sentinel.
Sentinel hosts: redistest1.mydomain.com:26379?ssl=true, redistest2.mydomain.com:26379?ssl=true, redistest3.mydomain.com:26379?ssl=true
Sentinel created
Host filter set.
Hostfilter: redistest1.mydomain.com:6379
Hostfilter: 10.100.60.72:6379
Hostfilter: 10.100.60.73:6379
RedisManager started.
Redis sentinel info: redistest primary: redistest1.mydomain.com:6379, replicas: 10.100.60.72:6379, 10.100.60.73:6379
Hostfilter: 10.100.60.72:6379
Hostfilter: 10.100.60.73:6379
Ping error with read only client: ServiceStack.Redis.RedisException: [14:23:47.626] Unable to Connect: sPort: 0, Error: One or more errors occurred.
(...)
---> System.AggregateException: One or more errors occurred. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

Is there anything more I can do on the Redis and/or Sentinel configuration side to ensure ServiceStack receives the actual host names of the Redis nodes in order for us to correctly validate the certificates used?

0

1 Answer 1

1

The ServiceStack log contains the necessary clue to what went wrong: Sentinel did exactly what it should, but the Redis backup nodes were being referred to by IP address.

Similar to how Sentinel can return a string containing the FQDN in the sentinel announce-ip configuration statement, Redis can do the same with the replica-announce-ip configuration statement.

The solution was to add the following line to the Redis configuration file on all Backup hosts:

replica-announce-ip servername.mydomain.com

Servername in this case of course is the hostname of the machine running this specific backup node.

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .