Nothing, in the appropriate circumstances. We do it for customers all the time.
The problem comes when you don't realise (or forget) that you've got IP address restrictions in place, and your last (or only) RDP-accessable IP address changes -- suddenly, you're locked out, and you can't fix it (because you're locked out).
We've solved the problem at work by adding our staff VPN ranges (which are RFC1918 and hence not likely to be forcibly renumbered) to the allowed IP ranges on all customer servers, so if a customer gets locked out they can always call us up to get the restrictions changed. We also have remote console access to everything (either via the VM host console, or iDRAC, depending on whether it's a VM or physical machine, accessable only via an out-of-band backdoor network with it's own, redundant, set of VPN-mediated access restrictions).
However, absent an out-of-band mechanism for (re-)gaining access, IP address restrictions always have that risk of locking yourself out completely (and, depending on your circumstances, possibly irretrievably).