It's (potentially) even worse than that -- you have to wait 24 hours after all of your authoritative servers have updated. The normal way for updates to happen is that you make a change to the zone on the primary server, and then each of the secondaries transfer the new zone data the next time they happen to check in with the primary. The check in frequency is controlled by the refresh interval in the zone's SOA record. Thus, in the worst case you'd have to wait the zone's refresh interval + the record's TTL.
You may also have to wait this long for the actual record changes. A 5-minute TTL won't do a lot of good if the secondaries only refresh every 6 hours. So you probably want to decrease the refresh interval on the zone as well for the period you want to be able to make quick changes.
Mind you, this may not apply to your setup. If you have a system that updates all authoritative servers together, this is not a problem (and I'm not familiar with Rackspace's DNS setup). But I'd recommend querying all of your authoritative servers individually (
dig server.example.com @secondaryserver.example.com) to make sure they have the new TTL before starting your 24-hour countdown.