When a domain name is registered, I understand that the nameservers for it are added in the root nameservers
This is a common misconception. The root nameservers only know about the "top level domain" (TLD) nameservers. Your
NS records were added to the registry for your TLD. The registry in turn publishes those
NS records to the "TLD nameservers" for your TLD.
If you have access to the commandline
dig utility, you can observe the full delegation path by running
dig +trace example.com NS.
. represents the root nameservers, and
com. is the registry managed zone for the "dot com" TLD.
Does this mean the ttls on the NS records control the caching?
Yes. There are two sets of
NS records that you need to be aware of. The first set is in your authoritative zone. The other set are in the delegation served by the TLD nameservers. (refer to the
dig +trace example.com output from the prior example)
When changing your nameserver IPs, the data needs to continue living on the old servers until the larger of these two TTLs sets have expired. The TLD nameservers typically have TTLs measured in days.
If this is confusing, think of it this way: you have no way of controlling which of these two TTLs that servers on the internet have seen most recently. You also have no control over whether DNS servers choose to prefer one set of
NS records over the other. The only thing you can to is ensure that you've waited for the longer of the two intervals.
Relatedly, from a recursive server point of view, are SOA records used for anything, or are they purely for handling zone transfers ?
The numeric intervals defined in the
SOA record are mostly used for communication between your master and secondary servers. The last field, SOA.MINIMUM, is special and used by recursive DNS servers when calculating how long they are allowed to cache the non-existence of a DNS record. (formula is
min(SOA TTL,SOA.MINIMUM), see RFC 2308)