How are the two sites connected to each other? Is there a WAN or dedicated site-to-site link, or are they both just connected to the Internet? If it's either of the previous two then the Internet won't be able to see traffic between the servers anyway, so your security concerns are greatly reduced.
If we're dealing with two separate but Internet-connected sites, then it's a little trickier, but you probably still don't need a full site-to-site VPN; some firewall/router rules to allow SSH/SCP and anything else you want to expose should do the trick. You can restrict access at both ends to only allow connections from the external IP addresses of the other site, assuming both sites have static external IPs.
EDIT (based on Reado's comment); Yes, it's possible. You can still do as I suggested in that second paragraph, although the data transmitted will still be in the clear and readable to anyone with network-level access to any point between server A and server B. If you want everything secure end-to-end, you're going to want either a site-to-site or server-to-server VPN tunnel. There are any number of VPN solutions that you could use; I'd suggest OpenVPN