4

My Setup:

There is only one physical machine in this setup, a Host System for Virtual Machines (VMs) with two network adapters.

One NIC (eth0) is connected to an internal network (LAN subnet, e.g. 10.x.x.x/24) and shall be used for internal traffic.

The other NIC (eth1) is connected to public internet (it has a public routable IP configured). This connection shall be used to port-forward public internet traffic to internal IPs of the VMs (incoming traffic) and to allow the VMs to access public internet (outgoing traffic) via NAT.

Virtual Machines use IP addresses in the LAN-Subnet (10.x.x.x/24, same as eth0)

I've got a bridge device (br0) configured for virtual network interfaces of the VMs (vnet0, vnet1, ...) and the LAN-NIC (eth0). That means:

  • br0 has an IP-Adress in the LAN subnet (10.x.x.x/24)
  • eth0 is added to the bridge
  • vnet0, vnet1, ... (used by the VMs) are dynamically added to the bridge

Problems

Communication within the LAN works fine. Also the VM-Host is accessable via the public IP and has internet access.

My problem is the NAT configuration to allow the VMs to access public internet, too.

I tried to use a simple (S)NAT rule:

iptables -t nat -I POSTROUTING -s 10.x.x.x/24 ! -d 10.x.x.x/24 -j SNAT --to-source y.y.y.102

Whereas y.y.y.102 is the public routable IP of the second NIC (eth1).

I found out that I need to enable "ip_forward" and "bridge-nf-call-iptables":

echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

Else the bridged packages won't be processed by iptables.

Now the packets from the VMs seem to go through the following Chains of iptables:

  • "FORWARD" (regular) - I accept them there (-j ACCEPT, counter goes up)
  • "PREROUTING" (nat) - I accept them there (policy ACCEPT, counter goes up)
  • "POSTROUTING" (nat) - They match the SNAT rule

But not all packets seem to arrive at PRE/POSTROUTING for any reason I couldn't figure out so far.

However, more interestingly tcpdump -i eth0 vs. tcpdump -i eth1 show that the packets (I tried to ping an external IP from within a VM) seem to be sent via the wrong interface eth0 (=LAN-NIC). Even the NAT rule was applied, so the source address was changed to the IP of the other NIC (eth1).

QUESTIONs:

How can I configure the system to output the NATed packets with the public IP as source address to be sent over the correct NIC (eth1)?

Do I somehow need to add eth1 to the bridge (br0)? If so, how do I assign the public IP address correctly? Usually the IP needs to be configured on the bridge device. Would I need to assign an alias adress to the bridge (public IP on br0:0)?

Configuration Details

The routing configuration on the host system:

# ip r
default via y.y.y.126 dev eth1
10.x.x.0/24 dev br0  proto kernel  scope link  src 10.x.x.11
y.y.y.96/27 dev eth1 proto kernel  scope link  src y.y.y.102
  • IP: y.y.y.126 is our router for public internet.
  • IP: y.y.y.102 is the public IP of the host machine
  • IP: 10.x.x.11 is the LAN IP of the host machine
  • SUBNET: 10.x.x.0/24 is the LAN
  • SUBNET: y.y.y.96/27 is the public IP subnet

NIC configuration:

# ifconfig
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.x.x.11  netmask 255.255.255.0  broadcast 10.x.x.255
        inet6 ####::###:####:####:####  prefixlen 64  scopeid 0x20<link>
        ether ##:##:##:##:##:##  txqueuelen 0  (Ethernet)
        RX packets 2139490  bytes 243693436 (232.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 29085  bytes 2398024 (2.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 ####::###:####:####:####  prefixlen 64  scopeid 0x20<link>
        ether ##:##:##:##:##:##  txqueuelen 1000  (Ethernet)
        RX packets 2521995  bytes 290600491 (277.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 383089  bytes 48876399 (46.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xdfa60000-dfa7ffff

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet y.y.y.102  netmask 255.255.255.224  broadcast y.y.y.127
        inet6 ####::###:####:####:####  prefixlen 64  scopeid 0x20<link>
        ether ##:##:##:##:##:##  txqueuelen 1000  (Ethernet)
        RX packets 2681476  bytes 597532550 (569.8 MiB)
        RX errors 0  dropped 130  overruns 0  frame 0
        TX packets 187755  bytes 21894113 (20.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xdfa00000-dfa1ffff

Bridge configuration:

# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.002590eb1900       no              eth0
                                                        vnet0

And iptables rules:

# iptables -vnL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  723  106K DROP       udp  --  *      *       y.y.y.0/24           0.0.0.0/0            udp spt:5404
  586 40052 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    5   420 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
    2   458 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            LOG flags 0 level 4
    2   458 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 1343  173K ACCEPT     tcp  --  *      *       10.x.x.2             0.0.0.0/0            tcp spt:3389
 1648  127K ACCEPT     tcp  --  *      *       0.0.0.0/0            10.x.x.2             tcp dpt:3389
   18  1040 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            LOG flags 0 level 4
   18  1040 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 525 packets, 84016 bytes)
 pkts bytes target     prot opt in     out     source               destination


# iptables -vnL -t nat
Chain PREROUTING (policy ACCEPT 13 packets, 1218 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 5 packets, 420 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 13 packets, 880 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 14 packets, 920 bytes)
 pkts bytes target     prot opt in     out     source               destination
    5   300 SNAT       all  --  *      *       10.x.x.0/24          !10.x.x.0/24           to:y.y.y.102

And here a captured NATed packet (ping from VM) on LAN interface card:

# tcpdump -i eth0
12:53:55.243350 IP y.y.y.102 > y.y.y.110: ICMP echo request, id 2, seq 5, length 40

Output of "ip rule":

# ip rule
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default
11
  • Can you show what happens in tcpdump -i eth1 when you try pinging some random external IP from the VM that owns vnet0? Also, check if /proc/sys/net/ipv4/conf/*/forwarding are set to 1. Jun 12, 2015 at 16:19
  • When pinging from the VM (using vnet0) to an external IP tcpdump -i eth1 doesn't show any related traffic. There were some STP broadcasts and some ARP request and other broadcast traffic... but nothing related to the ping. As I said, I see the ICMP packets leaving at eth0 (but obviosly no reply packet, as its the wrong NIC and theres no receipient reachable with that public IP on the LAN subnet); grep '' /proc/sys/net/ipv4/conf/*/forwarding shows that "forward" is set to 1 for every adapter. Any other ideas?
    – SDwarfs
    Jun 15, 2015 at 11:28
  • 1
    You will see them at eth0 because that interface is bridged with vnet0, there's nothing particularly strange about that part. But if you don't see them at eth1, that means that the kernel hasn't decided or hasn't succeeded to forward the packet to that interface. This could be because of iptables but also because of routing. You can check where it's getting stuck by adding LOG rules in nat and mangle table PREROUTING chains; by checking if you have alternate routing tables in ip rule; and by LOGging in mangle and filter FORWARD chains. Jun 15, 2015 at 14:47
  • @JosipRodin: I've no experience with the chains of 'mangle' nor 'filter'. What exactly are they supposed to do with the packets? --- The other chains have already some logging enabled (see section "Configuration Details" of the Question) without giving me reasonable insight.; Config-Details regarding "ip route" is also included in this section.
    – SDwarfs
    Jun 15, 2015 at 15:30
  • @JosipRodin: Sorry, you wrote "ip rule" (I read "ip route"). Added output of "ip rule" to the config section. Have no experience with "ip rule" either. Does that look as it should?
    – SDwarfs
    Jun 15, 2015 at 15:33

3 Answers 3

4
+50
  1. Check that your VMs have ip addresses on 10.x.x.x/24 (netmask 255.255.255.0)

  2. Set 10.x.x.11 (br0 ip address) as the default gateway of your VMs

  3. Enable ip forwarding on the physical host

  4. Enable SNAT with:

    iptables -t nat -A POSTROUTING -s 10.x.x.x/24 -o eth1 -j SNAT --to y.y.y.102
    
4
  • That did it! I had to set the IP of the VM host as standard gateway of the VMs! Thanks a lot! Could you also recommend some website/tutorial that explains this relation between default gateway and the routing mechanism (preferably at a detailed low level, so I actuall can understand how it really works)?
    – SDwarfs
    Jun 17, 2015 at 10:56
  • @StefanK. Ohh, now I see what the problem was. We were hoping for the kernel to forward packets using its routing table (Layer 3), but it likely wasn't seeing packets (Layer 3) but only frames (Layer 2) from the VMs, which were looking for the default gateway first. One can't NAT traffic to the default gateway - that has to go through directly. And the main reason you saw them in server iptables in that situation at all was that you explicitly turned on the option bridge-nf-call-iptables, which actually isn't the default. Jun 17, 2015 at 17:57
  • @StefanK. Note also that if you actually have other machines in the eth0 internal network and a separate default gateway on a different machine, you are bridging the vnet interfaces towards them and you can reach them, but if that default gateway also routes to elsewhere that eth1 doesn't, you can't use that implicitly on the VMs. Jun 17, 2015 at 17:58
  • @StefanK. Sorry, I have no tutorials/websites at hand. In a quick search on stack exchange sites with "routing" and "default gateway" I didn't see anything interesting. Maybe it could be a good subject for a separate question.. Jun 18, 2015 at 8:18
1

iptables -t nat -I POSTROUTING -s 10.x.x.x/24 ! -d 10.x.x.x/24 -j SNAT --to-source y.y.y.102

this must be changed to

iptables -t nat -I POSTROUTING --out-interface eth1 -j SNAT --to-source y.y.y.102

According to your first rule, only packages with destination to 10.x.x.x must be processed. So, what about traffic from outside to your network? (source - from worldwide, destination is your public IP :)

How can I configure the system to output the NATed packets with the public IP as source address to be sent over the correct NIC (eth1)?

read above. Just change the NAT rule.

Do I somehow need to add eth1 to the bridge (br0)? If so, how do I assign the public IP address correctly? Usually the IP needs to be configured on the bridge device. Would I need to assign an alias adress to the bridge (public IP on br0:0)?

By no means unless you're aware of what and why you're doing this. Keep internal and external interfaces separate. Allow routing only.

I have described by you configuration live (production) for more than 5 years. Working smoothly for 3 host servers and 25 VMs, including bridge links over openvpn tunnels.

6
  • 1
    Changed the rule accordingly. Unfortunately the rule doesn't match the packets anymore (and the source adress isn't mapped accordingly). The issue seems to be somewhere in the routing configuration itself. The kernel somehow wrongly decides to send the packets via eth0 ("[...]IN=br0 OUT=br0 PHYSIN=vnet0 PHYSOUT=eth0 [...]" within the logs). That is the reason why "--out-interface eth1" doesn't match the packets.
    – SDwarfs
    Jun 16, 2015 at 10:52
  • 1
    Anyway, Thank you for your answer. Knowing that this "should" work and is in production at least motivates to continue trying to make it working here. What Distribution/Version are you using for it? Any other idea?
    – SDwarfs
    Jun 16, 2015 at 14:21
  • 1
    Ubuntu 13.10 by 15.04 or 6 production servers. It does not matter actually, kernel net filter and iptables are pretty same on all Linux 2.6...3.x distros. Jun 16, 2015 at 22:15
  • Also, how are your VMs connected to the bridge? Any NATting performed by hypervisor? How do you assign them addresses? I use isc-dhcp-server for assigning the addresses and insert VMs as tap interfaces using scripts. You should review your VM scripts and change any kind of "-net -user" to "-net -tap" and add vm tap interfaces to the brigde. Again, according to your printouts - it's already done. Btw, check again. Jun 16, 2015 at 22:23
  • 1
    The issue is now solved by using the LAN-IP of the host machine inside the VMs as standard gateway, as suggested by Silvio. I had configured the LANs gateway, which seems to be wrong. But, thank you for your support and motivation!
    – SDwarfs
    Jun 17, 2015 at 11:01
0

Silvio's post helped me get a similar configuration working. In addition to his post, here are a couple of things that I needed to do as well.

  1. In newer versions of Linux kernels (Redhat 7 for example), you will need to enable the bridge kernel module:

    modprobe br_netfilter
    

and then to make that change persistent across server restarts, add the same line to a file named /etc/modules-load.d/.conf

  1. Once I enabled br_netfilter, I had to also enable iptable forwarding rules to the VM(s), e.g.:

    iptables -I FORWARD -d 10.x.x.x/24 -j ACCEPT
    iptables -I FORWARD -s 10.x.x.x/24 -j ACCEPT
    
  2. Instead of SNAT, I used a masquerade rule as I only needed a single route within a server hosting facility.

    iptables -t nat -A POSTROUTING -s <single-local-vm-ip>/32 -d <my-destination-subnet>/24 -p tcp -j MASQUERADE --to-ports 1024-65535
    

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .