How to configure dual homed server in order for both network segments to communicate?

Question

We want to add an additionnal network segment (LAN2) and need some machines on LAN1 to access ressources on LAN2. Some machines on LAN2 also need to access ressources on LAN1 and the WAN segment. Gateway between LAN1 and LAN2 is called "SRV-01", a dual homed Linux server.

I can't access LAN2 from LAN1 or LAN1 from LAN2 neither WAN from LAN2. I wonder how to make LAN1 hosts and LAN2 talk together and also how to allow access to the WAN from LAN2 machines.

Troubleshooting

I get the following replies using ping:

IP             (source)         ---> IP           (destination)    : Ping reply
---------------------------------------------------------------------------
192.168.5.33   (client on LAN1) ---> 192.168.5.8  (SRV-01:eth0)    : OK
192.168.5.33   (client on LAN1) ---> 10.0.2.1     (SRV-01:eth1)    : NO
192.168.5.33   (client on LAN1) ---> 10.0.2.2     (SRV-02:eth0)    : NO
*.*.*.*.*      (SRV-01)         ---> 192.168.5.33 (client on LAN1) : OK
*.*.*.*.*      (SRV-01)         ---> 10.0.2.2     (SRV-02:eth0)    : OK
10.0.2.2       (SRV-02)         ---> 10.0.2.1     (SRV-01:eth1)    : OK
10.0.2.2       (SRV-02)         ---> 192.168.5.8  (SRV-01:eth0)    : OK
10.0.2.2       (SRV-02)         ---> 192.168.5.33 (client on LAN1) : NO

Network topology

WAN --- (ISP Router) --- LAN1 --- (SRV-01) --- LAN2 --- (SRV-02) 

ISP Router

Internet service provider (ISP) router provides NAT (two network adapters) and act as a DHCP server for LAN1. We have no shell/admin access to this equipment.

• xxx.xx.xx.xx/xx (eth0, WAN)
• 192.168.5.4/24 (eth1, LAN1)

SRV-01 server

SRV-01 is a dual homed Ubuntu Server 14.04 (two network adapters) which would act as a gateway between LAN1 subnet and LAN2 subnet.

• 192.168.5.8/24 (eth0, LAN1)
• 10.0.2.1/24 (eth1, LAN2)

/etc/network/interfaces

# Loopback
auto lo
iface lo inet loopback
# LAN1
auto eth0
iface eth0 inet static
    address 192.168.5.8
    netmask 255.255.255.0
    network 192.168.5.0
    broadcast 192.168.5.255
    gateway 192.168.5.4
    dns-nameservers xxx.xxx.xxx.xxx
# LAN2
auto eth1
iface eth1 inet static
    address 10.0.2.1
    netmask 255.255.255.0
    network 10.0.2.0
    broadcast 10.0.2.255
    dns-nameservers xxx.xxx.xxx.xxx

Routing

~$ netstat -r
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
default         192.168.5.4     0.0.0.0         UG        0 0          0 eth0
10.0.2.0        *               255.255.255.0   U         0 0          0 eth1
localnet        *               255.255.255.0   U         0 0          0 eth0

~$ ip route show
default via 192.168.5.4 dev eth0 
10.0.2.0/24 dev eth1  proto kernel  scope link  src 10.0.2.1 
192.168.5.0/24  dev eth0  proto kernel  scope link  src 192.168.5.8

IP forwarding

~$ sudo sysctl -a | grep net.ipv4.ip_forward
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_use_pmtu = 0

Firewall

Ubuntu "Uncomplicated FireWall" (UFW) is disabled on server "SRV-01" and all iptables chains have a default policy set to ACCEPT target. Note that no single packet made his way to the FORWARD table yet...

~$ sudo ufw status
Status: inactive

~$ sudo iptables -L -v | grep policy
Chain INPUT     (policy ACCEPT 369 packets, 28278 bytes)
Chain FORWARD   (policy ACCEPT 0 packets, 0 bytes)
Chain OUTPUT    (policy ACCEPT 187 packets, 19740 bytes)

SRV-02 server

SRV-02 is an Ubuntu Server 14.04 with one network adapter.

• 10.0.2.2/24 (eth0, LAN2)

/etc/network/interfaces

# Loopback
auto lo
iface lo inet loopback
# LAN2
iface eth0 inet static
    address 10.0.2.2
    netmask 255.255.255.0
    network 10.0.2.0
    broadcast 10.0.2.255
    dns-nameservers xxx.xxx.xxx.xxx

Routing

~$ netstat -r:
Kernel IP routing table
Destination     Gateway     Genmask         Flags   MSS Window  irtt Iface
default         10.0.2.1    0.0.0.0         UG        0 0          0 eth0
10.0.2.0        *           255.255.255.0   U         0 0          0 eth0

~$ ip route show
default via 10.0.2.1 dev eth0 
10.0.2.0/24 dev eth0  proto kernel  scope link  src 10.0.2.2 

Firewall

Ubuntu "Uncomplicated FireWall" (UFW) is disabled on server "SRV-02" and all iptables chains have a default policy set to ACCEPT target.

~$ sudo ufw status
Status: inactive

~$ sudo iptables -L -v | grep policy
Chain INPUT     (policy ACCEPT 0 packets, 0 bytes)
Chain FORWARD   (policy ACCEPT 0 packets, 0 bytes)
Chain OUTPUT    (policy ACCEPT 0 packets, 0 bytes)

What am I missing here? I wonder what static route should be added to SRV-01 and also if default gateways are properly configured...

Answer 1

There are two problems with this setup:

1. The hosts on LAN1 know nothing about the LAN2 segment. When you ping a host on LAN1 (let's call it host1) from SRV-02, the packet will be routed through SRV-01 and will reach host1. However, host1 will send the reply to it's default gateway (ISP router) as it doesn't have a specific route to LAN2. (The ISP router will either a) also send it to it's default gateway as it also doesn't know about LAN2, or b) drop the packet as it comes from an unknown source not it's local LAN.)
2. When trying to reach WAN from LAN2, the packets will be routed through SRV-02 to ISP router where two situations are possible:
   • The router will not NAT translate the packet as the source of the packet (LAN2) is not it's local LAN (this is the more probable situation), or
   • The router will NAT translate the packet and send it to the Internet. However, when the reply comes and the destination is translated back to the LAN2 address, the packet will not be delivered as the ISP router doesn't have a route for that network. The packet will be sent incorrectly to the default gateway (ISP).

These issues could be fixed by adding a static route to LAN2 to ISP router and adding a source NAT configuration for LAN2 on SRV-01. However, that is not possible due to no admin access to the ISP router.

There are two solutions that get around it:

A. Make SRV-01 a full router for LAN1 and LAN2 hosts

• Add another network adapter to SRV-01 (making it 3 in total)
• Change the topology as follows:

.

WAN -> ISP router -> LAN1 -> SRV-01 +-> LAN3 (for hosts originally in LAN1)
                                    +-> LAN2 -> SRV-02

Basically, we're making SRV-01 a router for both LAN segments.

• This will require moving hosts originally in LAN1 to a new subnet LAN3 - let's say we use 10.0.1.0/24
• The network configuration of SRV-01 will need to be changed as follows:

/etc/network/interfaces:

# LAN1 - to ISP router
auto eth0
iface eth0 inet dhcp
# we can even use dhcp as the IP address is not really important
# - there are no more hosts on LAN1 apart from ISP router and SRV-01

# LAN3 - for hosts originally in LAN1
iface eth1
    address 10.0.1.1
    netmask 255.255.255.0

# LAN2
iface eth2
    address 10.0.2.1
    netmask 255.255.255.0

iptables rules to make WAN access work:

iptables -t nat -A POSTROUTING -o eth0 -s 10.0.1.0/24 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth0 -s 10.0.2.0/24 -j MASQUERADE

Alternatively, if you choose to keep the static IP address on SRV-01 on eth0 the rules could be changed (although MASQUERADE would still work):

iptables -t nat -A POSTROUTING -o eth0 -s 10.0.1.0/24 -j SNAT --to-source 192.168.5.8
iptables -t nat -A POSTROUTING -o eth0 -s 10.0.2.0/24 -j SNAT --to-source 192.168.5.8

• DHCP will need to be configured on SRV-01 on eth1 (LAN3, for hosts originally on LAN1), and possibly on eth2 (LAN2) as well if required. (In both cases the gateway will be the local address of eth1 or eth2 respectively, but that goes without saying :)

This will make communication possible between LAN3 and LAN2 (via SRV-01 which is the default gateway for both). WAN access will also work from both LAN3 and LAN2 thanks to the double source NAT.

B. Make SRV-01 a DHCP server for LAN1

This approach is not as clean as above but is slightly simpler. It assumes you are able to disable DHCP on ISP router

• Disable DHCP on ISP router
• Set up DHCP for LAN1 on SRV-01 and make SRV-01 (192.168.5.8) the default gateway for LAN1
• Set up source NAT translation for LAN2 on SRV-01 so that the WAN access works from LAN2:

.

iptables -t nat -A POSTROUTING -o eth0 -s 10.0.2.0/24 -d 192.168.5.4 -j SNAT --to-source 192.168.5.8
iptables -t nat -A POSTROUTING -o eth0 -s 10.0.2.0/24 ! -d 192.168.5.0/24 -j SNAT --to-source 192.168.5.8

The first line enables SNAT so that LAN2 hosts can access the ISP router itself and the second line disables SNAT for LAN2-LAN1 access.

Again, this approach is not as clean as the one above as there are two routers in the same subnet (SRV-01, ISP router). When I used this approach myself I noticed my second router (SRV-01 in this scenario) would send ICMP redirects to the ISP router as it would see that the client (host on LAN1) and the upstream router (ISP router) are on the same LAN. This might not be desired as network policies implemented on SRV-01 could be circumvented.

Hope that helps.