As it turns out, for reasons still unknown, my correctly-configured networking service on my web server was bringing up its IP address with a /8
netmask, instead of the (in my case) correct netmask of /24
.
Manually removing the offending entry from my server’s routing table and adding the correct entry resolved the issue, and Googlebot is now happily visiting my web server again:
root@web:/# ip addr show eth0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
inet 192.0.2.7/8 brd 192.255.255.255 scope global eth0
root@web:/# ip addr add 192.0.2.7/24 dev eth0
root@web:/# ip addr del 192.0.2.7/8 dev eth0
root@web:/# ip addr show eth0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
inet 192.0.2.7/24 scope global eth0
(IP addresses changed to protect the innocent. Adding the correct entry before removing the incorrect one helps keep your SSH session open if you’re connected via that address.)