I don't think that would be practical, due to a custom rule could be created and you wouldn't be able to cover all of those rules. It is possible to use group policy to not apply local rules, but that may break something.
An alternative would be to create two deny rules. The first rule denies remote IP address range 0.0.0.0-. The second rule denies remote IP address range . For example, to allow IP address 10.1.2.3:
TCP/3389 Deny rule 1: Deny remote address range: 0.0.0.0-10.1.2.2
TCP/3389 Deny rule 2: Deny remote address range: 10.1.2.4-255.255.255.255
And of course you should create an TCP/3389 allow rule for your one IP address.
You should also create two deny rules for UDP/3389, as modern Windows operating systems may use UDP, unless you have group policy configured for remote desktop to only use TCP. Or just create one UDP/3389 firewall deny rule for all addresses if you don't want or need it.