I have a bizzarre networking problem.
There is a network, a single IP subnet: 192.168.60.0/24.
The network has two Internet-connected firewalls (running ForeFront TMG 2010), 192.168.60.253 (production) and 192.168.60.240 (development); this is a necessity because there are two environment and each of them needs its own Internet-publishing rules; however, for historical reasons, they share the same IP subnet, and this can't be easily changed. Thus, production computers use 192.168.60.253 as their default gateway, while development computers use 192.168.60.240. Everything works, as long as no other networks are involved.
But, of course, there is another network (actually more than one, but let's simplify things)connected to a third NIC on the production firewall, namely 192.168.100.0/24. Computers in this network use that firewall as their default gateway. They can talk with production computers in the main network, and vice-versa.
When computers in the 192.168.100.0/24 network try to talk with development computers in the main network, this doesn't work, because their default gateway (the develompent firewall) doesn't know how to send packets back. This is expected. So I added a permanent static route to the development firewall to tell it "when you need to send a packet to 192.168.100.0/24, send it to 192.168.60.253"; I also defined the network in the networking configuration of TMG end enabled all traffic between the two networks.
Now, the really strange part: pinging between the two networks works, but any other protocol doesn't. HTTP, RDP, SMB... nothing goes through, although routing seems to be working and firewall policies are open for everything.
What's happening, why, and how can I fix it (besides manually adding static routes to each development server, which I'd like to avoid)?
I examined the traffic logs in the development firewall, and it looks like TCP packets are being dropped because, from that firewall's point of view, they are not related to any open connection. This actually makes sense, because when a connection is started, the initial packet of the TCP handshake goes through the production firewall, while the answer tries to come back through the development firewall... and TMG drops it because it never saw the first packet at all. This also explains why TCP doesn't work while ICMP (and presumably UDP) does.
But still: how can I fix this and make this asymmetric routing work, if it can at all be done?