From my reading of this page, it appears that ntp doesn't use the INADDR_ANY
0.0.0.0 address exclusively partly for security reasons, and partly for authentication reasons.
First port 123, is below 1024, and so is considered a privileged port, and only root can bind to that port. Ntp is typically set to drop privileges after it is started. From what I understand from the mail lists, and the article once the privileges are dropped can't open a socket to reply from correct source port of 123, so ntp opens up sockets for every assigned address before it drops privileges.
From what I have read some of the authentication mechanisms for ntp basically require that the source and destination port be 123, and nothing else.
The matter isn't entirely clear. See the section about the wildcard address
0.0.0.0, it is opened by ntpd for some reason, but from the comments should never actually be used, except possible in some special rare cases, that the devs aren't entirely sure about, but, they don't want to remove the socket, just in case they break things.
Note that normally ntpd should not be accepting packets on the wildcard addresses since there are a number of problems if you do so including sending return packets on a different address from the sender's requested address.
DannyMayer - 27 Apr 2009
I think the main answer to your question is in the above comment here.