I always get nervous when editing iptables as I know how simple it can be to end up blocking all traffic to the server, a rather large issue when your server is sitting on the cloud.
Would these be the correct series of commands?
iptables -A INPUT -p tcp --dport 123 -s 1.2.3.4 -j ACCEPT
iptables -A INPUT -p tcp --dport 123 -s 5.6.7.8 -j ACCEPT
iptables -A INPUT -p tcp --dport 123 -j DROP
iptables-save