Let's say we don't want our server (any program running on it) to talk to example.com (any IP address that this DNS name can happen to be resolved into) and anything like *.example.com etc. Both inbound requests coming from those domains and outbound requests made by programs running on our server trying to reach the forbidden domains are to be blocked. What is the right way to set this up?
First of all - rethink it, it will bring you more problems that successes.
You cannot use DNS in firewall, resp. you cannot use DNS to check "Is this IP belongs to domain?" from many reasons, biggest is non consistent recursive DNS. You can has only one PTR record for IP and this records are managed with IP owner. Not so many domains are really owners of their IPs - they just use some service provider like shared hosting, cloud provider and so on - so if you will try to resolve IP to domain, you will get those provider, not domain.
In example serverfault.com has IP 22.214.171.124. But when you will trying to detect if this IP belongs to serverfault.com, you will fail, because this IP belongs to CloudFlare.net and there is no connection between CloudFlare.net and serverfault.com.
If your "not wanted" domain has their own IP subnet, you can use
whois tool to detect their assigned subnet and block that subnet. In case upper, you can block whole CloudFlare, but you will block everyone who is using CloudFlare, what is not what you want.
Next way is hacking DNS and steal zone files for that domain (use AXFR and try to inicialize zone transfer - see this question on SuperUser). Some DNS servers will send whole zone file to anyone who will ask, but most of them will send it just to allowed servers. If you will get that file, you can parse it and detect used IPs. But again - if this is some CDN or shared hosting, you will block anything on that server.
Firewall or DNS or Proxy.
You can configure your perimeter firewall and blacklist/block all incoming/outgoing traffic from and to the domain's associated ips. And of course you can also do it in each server level firewall. You didn't say what kind of firewall you have in your network. Some firewall distribution has different options to achieve something similar. But you can achieve this with iptables in a linux box. For example, pfSense offers the ability to create
aliasand use it in firewall rules. See here: Blocking websites.
Another possiblity is to create a list of ips by querying dns for the domain, save it to a file and feed it to the iptables/firwall to block. A custom script can be used to query and generate the ip list and keep it upto date.
Local DNS servers can also be used to achieve this. You can configure a dns forwarder or a dns proxy, and point the domain to a fake ip or a firewall ip, (kind of dns spoofing). For example, with software like
dnsmasqit is very easy to point a domain to an ip that you want to. This is only effective if all your dns queries for your network is forwarded to this dns server and are not allowed to send directly to internet. Blocking servers with dnsmasq
Another possiblity is to use proxy server or content filter to block access to certain domain, which is also easy. The thing is, it only works for http or https protocol.
Keep in mind that creating "subdomains" that will resolve to every possible IP address (i.e. 126.96.36.199.example.com, 192.168.1.1.example.com, 254.254.254.254.example.com etc.) is a pretty easy task so it's impossible to fully block incoming connections without having some kind of finite list of host names.
Blocking outgoing on the other hand can be very simple - just add
to your hosts file (i.e.
/etc/hosts) or a wildcard DNS record (
*.example.com A 127.0.0.1) to your local DNS server and it will be enough to block outgoing connections (provided they use domain name to connect)
P.S. It's possible to block connections to certain physical server though, by using something like a GeoIP database to block all IPs coming from certain AS