How is it possible to have a default gateway outside of the network range of any interface?

Question

I always thought that the default gateways has to be on the same network as the server/client trying to connect to the internet. However today I rented a server from Hetzner and the server is able to connect to the internet even though, as it seems to me, the default gateway is definitely not connected to the same network. Also the netmask seems to be set to /32.

root@test:~# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 138.201.187.120  netmask 255.255.255.255  broadcast 138.201.187.120
        inet6 fe80::9400:ff:fe3b:eece  prefixlen 64  scopeid 0x20<link>
        inet6 2a01:4f8:c17:69b0::1  prefixlen 64  scopeid 0x0<global>
        ether 96:00:00:3b:ee:ce  txqueuelen 1000  (Ethernet)
        RX packets 1166  bytes 638244 (638.2 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 653  bytes 92539 (92.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 112  bytes 8824 (8.8 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 112  bytes 8824 (8.8 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@test:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.31.1.1      0.0.0.0         UG    0      0        0 eth0
172.31.1.1      0.0.0.0         255.255.255.255 UH    0      0        0 eth0
root@test:~# ip route
default via 172.31.1.1 dev eth0
172.31.1.1 dev eth0 scope link
root@test:~# ping ipv4.google.com
PING ipv4.l.google.com (172.217.21.238) 56(84) bytes of data.
64 bytes from fra16s13-in-f14.1e100.net (172.217.21.238): icmp_seq=1 ttl=54 time=4.83 ms
64 bytes from fra16s13-in-f14.1e100.net (172.217.21.238): icmp_seq=2 ttl=54 time=4.96 ms
64 bytes from fra16s13-in-f14.1e100.net (172.217.21.238): icmp_seq=3 ttl=54 time=5.01 ms
^C
--- ipv4.l.google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 4.837/4.938/5.010/0.073 ms
root@test:~#
root@test:~# traceroute ipv4.google.com
traceroute to ipv4.google.com (216.58.207.78), 30 hops max, 60 byte packets
 1  _gateway (172.31.1.1)  0.139 ms  0.091 ms  0.062 ms
 2  12643.your-cloud.host (136.243.182.17)  0.320 ms  0.103 ms  0.075 ms
 3  * * *
 4  213-239-251-233.clients.your-server.de (213.239.251.233)  1.479 ms 213-239-251-237.clients.your-server.de (213.239.251.237)  0.722 ms  1.320 ms
 5  core21.fsn1.hetzner.com (213.239.239.125)  0.417 ms  0.310 ms  0.278 ms
 6  core4.fra.hetzner.com (213.239.245.18)  5.536 ms core4.fra.hetzner.com (213.239.245.14)  4.880 ms core0.fra.hetzner.com (213.239.252.33)  5.539 ms
 7  72.14.218.94 (72.14.218.94)  5.094 ms  5.085 ms  5.113 ms
 8  108.170.251.193 (108.170.251.193)  5.018 ms * 108.170.252.65 (108.170.252.65)  6.311 ms
 9  108.170.235.246 (108.170.235.246)  7.428 ms 72.14.232.50 (72.14.232.50)  4.903 ms 72.14.234.227 (72.14.234.227)  4.951 ms
10  108.170.252.18 (108.170.252.18)  5.287 ms fra16s25-in-f14.1e100.net (216.58.207.78)  4.804 ms 108.170.251.145 (108.170.251.145)  5.646 ms
root@test:~#

Thanks!

Answer 1

To be exact, the gateway needs to be on the same physical network as the computer, as reaching it (or reaching any computer on the network, really) requires ARP packages to be sent. For this, any local network requires a route entry, which defines "on-link" computers. Here is what a route entry for a more common network setup would look like:

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.193.254 0.0.0.0         UG    0      0        0 enp0s3
192.168.193.0   0.0.0.0         255.255.255.0   U     0      0        0 enp0s3

As you see from the second line, it is well defined how to reach the gateway server. The Gateway column is 0.0.0.0, which means "use no gateway for these addresses", so it defines those addresses to be directly available. Unfortunately, the output of route -n is not intuitive in this sense, but if you use ip route instead, you would see a somewhat more straightforward representation:

default via 192.168.193.254 dev enp0s3
192.168.193.0/24 dev enp0s3 scope link

Here, dev enp0s3 scope link means "this is on-link with the enp0s3 device, so it is directly accessible through it". Since the gateway itself is in that subnet, the computer knows it can be reached directly, so if needed, it knows how to route packages to the outside world.

If the second entry would not be there, the gw would be unreachable, even if it is on the same network.

In your case, the second line of the route entry serves that purpose, although it is for a single host, not for a network:

172.31.1.1      0.0.0.0         255.255.255.255 UH    0      0        0 eth0

or, as the output or ip route shows:

172.31.1.1 dev eth0 scope link

So the solution is that your computer and the gateway are probably on the same physical network, and your server is told how to reach for the gateway. From this point it doesn't matter if they share the same logical network or not.

Also, it doesn't matter what the netmask is, as long as the routing table contains entries to the computers. Setting the netmask only defines how to populate the routing table, but if you have a /24 subnet entry in the routing table (like in my first example), and set the netmask to /32, the server will happily reach out for each computer defined in the routing table.

This works in the opposite direction as well: if you have a /24 netmask, but not "on-link" routing entries, then the local network will be inaccessible.