We recently got done doing a massive IP scheme change at work and since then I've noticed a few oddities. First, our setup:
IPCop firewall which has - Wireless network (completely segregated) - Firewalled DMZ - Internal network [eth0] (10.10.10.x) - OpenVPN network [tun0] (172.16.20.x) - Internet
Before the IP change, our OpenVPN network could talk to our internal network without any problem. Since the change we have a few servers that no longer want to talk across the boundary properly.
For example, I will connect to the VPN with 172.16.20.10 and try to talk to a server at 10.10.10.19. Running tcpdump I see the packets leave my machine (running wireshark), travel through tun0, travel through eth0, hit 10.10.10.19, and nothing comes back. Other servers work fine on the same network (10.10.10.8, for example, works fine) and I cannot find a common piece between them on what would cause some servers to just stop responding to some requests.
I checked things like hosts.(allow|deny), pf/iptables, etc, that would normally filter traffic but none of them have exclusion rules that would stop traffic like this.
Any ideas on where to go from here?
The boxes that were having the problem did indeed have bad gateways that got overlooked. Made the changes in the appropriate places and everything seems to be working!