Why does my NAS server not work on both Production and Isolated Lab Subnets?

Question

In our VMWare infrastructure, I have a production network (10.1.1.0/24 subnet) and an isolated virtual lab network (also 10.1.1.0/24) which is created by Veeam, with the Veeam proxy appliance sitting between them.

I created a NAS VM (FreeNAS 9.10) with two virtual NICs, and I wanted to connect one to the production network, and one to the isolated lab network, for easy transfer of large files between the two. However, this does not work. It seems that whichever network is connected first works (I can ping the VM, pull up the Web GUI, pull up the Windows Share) but when I connect the second network, the VM cannot be accessed by the second network. Even if I set a static IP on both, and use a different 10.1.1.0/24 IP address on each virtual NIC, only the first network connected seems to work.

In my primitive understanding of networks, I'm thinking maybe the traffic INTO the NAS on that second network is working, but the NAS is trying to reply on 10.1.1.0/24 and thinks it should do so on the other virtual NIC? Maybe this setup is impossible? Although, that seems unlikely, since somehow the Busy Box virtual appliance/router/proxy that Veeam creates can handle a similar setup.

Answer 1

Having the same network on both NICs in your NAS is your problem. It's confused.

Answer 2

A good practise is to use different network ranges for production and lab. For example, you use 10.1.1.0/24 subnet for production and 10.1.2.0/24 subnet for lab.

Update : According to your needs, you want the NAS to be seen with the same IP adress in the prod subnet and the lab subnet. What I would do is set up NAT (on a small fw or linux box or linux VM) in front of the NAS. Therefore prod and lab with have same IP adress while still being isolated.

    
+------------------+
| PROD 10.1.1.0/24 |
+------------------+
        |
+----------------------+
| NAS                  |
| re0 10.1.1.50/24     |
| re1 192.168.42.50/24 |
+----------------------+
        |
+----------------------+
| VMFW                 |
| eth0 192.168.42.50/24|
| eth1 10.1.1.50/24   |
+----------------------+
        |
+----------------------+
| LAB 10.1.1.0/24      |
+----------------------+

edit don't forget a few network magic to send all network destinated to 10.1.1.50 interesting ports to 192.168.42.50.

Answer 3

I think @bgtvfr has a very good, very close answer. I've sort of given up on this whole idea for now, but I know it is possible. The Veeam Virtual Lab proxy appliance does this for us on our network, and works well, and after doing some research and looking at all of the startup scripts on this (BusyBox Linux) device, I believe the key piece of configuration I'm missing is using something like netfilter to mark packets.

E.g. from https://unix.stackexchange.com/a/4421

The following commands create an alternate routing table via eth1 for packets that have the mark 1 (except packets to localhost). The ip command is from the iproute2 suite

ip rule add fwmark 1 table 1 
ip route add 127.0.0.0/0 table 1 dev lo
ip route add 0.0.0.0/0 table 1 dev eth1 

The other half of the job is recognizing packets that must get the mark 1; then use iptables -t mangle -A OUTPUT … -j MARK --set-mark 1 on these packets to have them routed through routing table 1. I think the following should do it (replace 1.2.3.4 by the address of the non-default-route interface):

iptables -t mangle -A OUTPUT -m conntrack --ctorigdst 1.2.3.4 -j MARK --set-mark 1 

I'm not sure if that's enough, maybe another rule is needed on the incoming packets to tell the conntrack module to track them.

Here is the clue I found in the configuration / init file for the Veeam Proxy appliance, where eth0 and eth1 are both 10.1.1.0/24 subnet - eth0 being my production network and eth1 the virtual test lab: I'm just too green with netfilter/iptables, but seems like maybe this is marking packets that come in on the virtual lab interface so that they go back out on the same interface rather than the default gateway interface?