1

I am trying to setup two micro-instances behind an internal ELB which will handle mundane tasks such as central logging and NTP queries, but come to realize that ELB doesn't support UDP forwarding. After some searching I came across this article, and looked into it, but according to AWS docs, you cannot setup health checks on nodes within a private zone.

So my question is, how do I enable the ability to host a single NTP server with fault tolerance (ELB or Route-53, or ?) so local nodes can reference it, vs polluting internet channels with requests? Do I need to manage my own fault tolerance mechanism via VIP and keepalived?

Improve this question

1 Answer 1

Reset to default
2

Don't try to load balance or otherwise "HA" your time servers. That goes contrary to how NTP is designed to work -- by comparing the servers against each other. DNS based failover is a non-starter, too. There is also no point in a naive health check of ntpd, because the daemon can be running and happy, despite having lost its mind and giving out invalid time signals.

Note that ntpd will resolve the name of a designated time server into an IP address when it starts, and will not attempt to re-resolve that information once it is running (unless/until you tell it to re-read/reload the configuration file).

This means that if you think you're getting higher reliability by having just one server name but multiple IP addresses (which might point to multiple machines), you should think again. When you're managing a very large group of NTP servers (e.g., pool.ntp.org) there are advantages to this, but for anything smaller, you're almost certainly not going to get the behaviour you're probably thinking about.

Also, don't try to give multiple machines the same IP address, or hide them behind a load-balancing device, either - that will really confuse ntpd and make the situation far worse.

It's much better to have a specified set of servers, each with their own unique IP address, and configure the clients to connect to multiple servers in this set and then let the clients deal with the issues of what happens when one or more of their servers becomes unreachable or unreliable.

http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPServers#Section_5.3.6.

Configure the servers individually and set the clients to use all of them... all 4 of them, since 4 is considered the low bound for an appropriate number of servers, as it leaves you with what is still a high quality time signal from three servers, if one of the servers should lose its mind. One bad out of three doesn't allow the one bad server to be effectively ignored.

http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPServers#Section_5.3.3.

From my perspective, the NAT instances used in VPC are appropriate candidates for running your time servers.

Or... consider that once ntpd stabilizes, it generates less and less traffic, and don't worry too much about the relatively negligible amount of Internet traffic (but of course don't configure the servers all to connect directly to stratum 1 sources).

Improve this answer
2
  • It's not about the traffic at all. It's about ensuring that the nodes are all in sync, so if an AZ is down, I won't end up with nodes running +/- 5 minutes which as you know could wreck havoc with analytics. I suppose I could just add 2 dns entries to the domain, and update the ntpd.confs to reference them, and just run off the two nat servers (1 in each AZ). Thanks.
    – Mike Purcell
    Oct 22, 2015 at 2:57
  • for having islands staying intact time-wise, look at orphan mode. if you have serious needs to have time intact.
    – Florian Heigl
    Dec 11, 2016 at 16:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .