Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a couple router/firewall boxes (pfSense, TMG 2010, ISA 2006) on my network that are stateful. Right now they all have an interface on the same subnet as most end user devices and servers. I will be making some changes and putting some servers on their own subnets behind these firewalls so I'm wondering if I should set up one dedicated subnet for the routers to route packets to each other through. There are no routing protocols, only static routes.

I'm trying to avoid asynchronous routing which can be a problem for stateful firewalls since traffic flows in a different path into and out of the network. If traffic flows back through a different path and the firewall in that path does not have a record in the state table then the traffic could get blocked.

My basic question is this: is this an ideal way to approach this problem? Why or why not? I have not been able to find much in terms of best practices but this approach would leave only one router on each subnet so I would avoid the current situation of different machines having different default gateways.


Router 1              Router 2              Router 3 ------ ------ ------ All other devices
       |                     |                      |
       V                     V                      V


Router 1              Router 2              Router 3                                              ------ All other devices ----- ----- ------ Routers/Firewalls only
       |                     |                      |
       V                     V                      V    
share|improve this question
I'm assuming is a typo and should be /24? – fukawi2 Jul 8 '13 at 1:46
up vote 1 down vote accepted

Following on from my comment, something like this

      +----------+    +----------+   +----------+
      | Router 1 |    | Router 2 |   | Router 3 |
      +-------+--+    +----+-----+   +--+-------+
              |            |            |
              |            |            |
              |            |            |
           |           Router A           +-------------+
           +-+---------+---------------+--+             |
             |         |               |                |
             |         |               |                |
             |         |               |                |
+------------v-+ +-----v-------+ +-----v-------+ +------v------+
|| || || ||
+--------------+ +-------------+ +-------------+ +-------------+

Router 1 =

Router 2 =

Router 3 =

Router A =

This way, each of the networks at the bottom only has 1 router, and therefore 1 default route. The edge routers only need 1 internal route to access the internal subnets.

The internal router does become more complex because it needs to be aware of the multiple upstream routers, and keep track of connections to avoid the async routing. The benefits are worth it I believe: All the complexity is contained to that router, with the rest kept simple. And you can have full control over the multiple connections on that host. For example, you can NAT traffic from all 3 connections to the same internal server, but the server doesn't need to know any of that, the internal server will track each connection and route the traffic appropriately to avoid async.

This is very similar to the setup at my work, except we only have 2 upstream connections. Router A is a pair of Linux boxes running in H/A. Tracking of the connections is done using Policy Based Routing. The best guide I've found for PBR is:

share|improve this answer
Okay I see where you're going with this and it makes sense. I don't have multiple ISPs though; just one with a /28. I think your diagram would work except for the fact that each router has different functions. My Router A would be pfSense and I want to publish MS stuff behind TMG2010 which would be Router 1. So in this diagram how would I publish behind Router 1 for users on and external clients? – Oliver Jul 9 '13 at 1:44
Why do you have 3 routers for the same /28? And what do you mean "Publish"? – fukawi2 Jul 9 '13 at 5:01
I mean "publish" as used in MS TMG which means the client connects and authenticates to TMG instead of directly to the Exchange or Sharepoint server. PfSense is my main router but can't perform the same type of authentication services at the application layer so I use ISA and TMG on the few other public IPs I have. – Oliver Jul 9 '13 at 12:32
here's a link to Microsoft's definition of "publishing" a Sharepoint server – Oliver Jul 9 '13 at 14:40
Ohhhh, well I'm out then. I have no idea about that gear. Sorry. – fukawi2 Jul 9 '13 at 23:05

I would say you're introducing an extra level (literally) of complexity.

I'm assuming your async routing issues come from the situation eg sends a packet to but doesn't have a route for it via so sends it via it's default at where it gets redirected to The reply packet then goes directly from to the original source, so only see's the client to server packets, not the server to client packets.

In my environments, I avoid the firewall issue with async routing by adding a rule to allow bounce routing (anything in and out the same interface == same layer 2 == allow). You're not introducing any security issues since the src and dst of a packet are on the same layer 2 and can talk directly to each other anyway, you're just "encouraging" performance issues by routing sub-optimally.

share|improve this answer
Fukawi2, yes your description of the problem is correct in general although I simplified it a bit. I will attempt to explain the actual problem I've had which is not so much rules. – Oliver Jul 8 '13 at 22:44
sorry didn't know "Enter" sent the comment. Assume both Router1 and Router2 also have publicly addressable IPs on another interface and NAT is enabled. If Server A with IP is published behind Router2 but has Router1 as it's def GW then a packet coming in from the internet via Router2 will be replied to via Router1 and the packet will be blocked by the firewall on the far end. In this case I need to set Server A to use Router2 as the def GW and add static routes on Server A to networks behind the other routers and this seems too complex. – Oliver Jul 8 '13 at 22:53
Ah I see. I think your original suggestion is reasonable then. Personally, I'd probably do it slightly differently, in that I'd isolate the 3 routers (to handle all the public in/out traffic) and then have another one to handle all the internal traffic. I'll post another answer with a better description. – fukawi2 Jul 8 '13 at 23:02

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.