You should always have a "deny all" default rule in your firewall. Thus if things aren't explicitly allowed, they will be denied. It would be a rare thing for a firewall to not have this rule configured by default, and if you don't then you need to create it.
Assuming that is the case, you need only open the appropriate ports to the appropriate IP. You don't have to also create deny rules for every other IP that might try to connect.
You should always test your firewall configuration with something like Shields Up (if it's on the public net) and/or nmap (which you run locally).