3

I would like to setup a 1:1 NAT + reflection with iptables, so really forward everything which arrives on that one public IP to my internal VM.

Background Info: I have a dedicated server running proxmox with a NAT based network configuration (see below). One NIC one public main IP (94.x.x.A) and an extra ordered second public IP (94.x.x.B) on the same interface. I also have a virtualised opnsense/pfsense VM which uses vmbr10 and vmbr11 as WAN and LAN. Spoiler, I tried the 1:1 Nat via opnsense/pfsense already. I really tried everything in here but couldn't make big blue button work. I thought, okay maybe all the nat'ing through vmbr10 (WAN) into vmbr11 (LAN) and through opnsense/pfsense itself causes to much trouble for BBB. Therefore the idea was to make the routing easier.

auto enp0s31f6
iface enp0s31f6 inet static
  address 94.x.x.A
  netmask 255.255.255.192
  gateway 94.x.x.x
  mtu 1500
  post-up echo 1 > /proc/sys/net/ipv4/ip_forward
  post-up echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
  up ip addr add 94.x.x.B/26 dev $IFACE label $IFACE:0
  down ip addr del 94.x.x.B/26 dev $IFACE label $IFACE:0
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p tcp -d 94.x.x.A -m multiport ! --dport 22,8006,179 -j DNAT --to 10.10.10.2
  post-up iptables -t nat -A PREROUTING -i enp0s31f6 -p udp -d 94.x.x.A -m multiport ! --dport 5405:5412,4789 -j DNAT --to 10.10.10.2

Now I created another linux bridge (vmbr0) and moved this one VM with Big Blue Button out of vmbr11, not going through opnsense/pfsense anymore and I would really like to forward everything to it. Iptables should absolutely not care about anything, blindly forward everything arriving at that second public IP 94.x.x.B to my local BBB VM. So like the VM itself is directly connected to the internet. It basically owns that IP.

I tried to do it with 1:1 NAT and reflection, but no chance I could get it to work. I tried every iptables command I could find in the whole www. Simply impossible, there are two commands which simply fail all the time, no Mather what I do. They are executed on that VM itself:

docker exec -it bbb-docker-greenlight-1 bundle exec rake conf:check

Checking environment: Passed
Checking Connection: Failed
Error connecting to BigBlueButton server - Failed to open TCP connection to bbb.my-domain.com:443 (Connection refused - connect(2) for "bbb.domain-domain.com" port 443)

and curl --trace-ascii - -k https://94.x.x.B:443/bigbluebutton/api which immediately shows:

== Info:   Trying 94.x.x.B:443...
== Info: connect to 94.x.x.B port 443 failed: Connection refused
== Info: Failed to connect to 94.x.x.B port 443: Connection refused
== Info: Closing connection 0
curl: (7) Failed to connect to 94.x.x.B port 443: Connection refused

So I have the feeling when the VM itself tries to connect to that public IP from within local vmbr0, the reflection part is not working, something is wrong. The curl command is failing immediately, like nothing listens on 443. But the nginx docker container is running, I can access the main page of the bbb service when entering the local VM IP in the browser when connected to my VPN.

Can someone with good iptable knowledge please help me to forward everything to that one VM and setup reflection? I don't think that split dns is working, as it seems that in the whole BBB docker construct, they use the public IP and not the domain name.




Collections of commands, I tried mostly whole blocks, commented out, tried a new block etc. plus single commands from each block. This topic gives me already nightmares, so please help me someone. Thanks

auto vmbr0
iface vmbr0 inet static
        address  10.2.1.1/30
        bridge-ports none
        bridge-stp off
        bridge-fd 0

        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward

        post-up iptables -t nat -A PREROUTING -i enp0s31f6 -d 94.x.x.B -j DNAT --to 10.2.1.2
        post-down iptables -t nat -D PREROUTING -i enp0s31f6 -d 94.x.x.B -j DNAT --to 10.2.1.2

        post-up   iptables -t nat -A POSTROUTING -o enp0s31f6 -s 10.2.1.2 -j SNAT --to-source 94.x.x.B
        post-down iptables -t nat -D POSTROUTING -o enp0s31f6 -s 10.2.1.2 -j SNAT --to-source 94.x.x.B
        #post-up   iptables -t nat -A PREROUTING -i enp0s31f6 -d 94.x.x.B -j DNAT --to-destination 10.2.1.2
        #post-down iptables -t nat -D PREROUTING -i enp0s31f6 -d 94.x.x.B -j DNAT --to-destination 10.2.1.2
        #post-up   iptables -t nat -A PREROUTING -i vmbr0 -s 10.2.1.0/30 -d 94.x.x.B -j DNAT --to-destination 10.2.1.2
        #post-down iptables -t nat -D PREROUTING -i vmbr0 -s 10.2.1.0/30 -d 94.x.x.B -j DNAT --to-destination 10.2.1.2
        #post-up   iptables -t nat -A POSTROUTING -o vmbr0 -s 10.2.1.0/30 -d 10.2.1.2 -j SNAT --to-source 10.2.1.1
        #post-down iptables -t nat -D POSTROUTING -o vmbr0 -s 10.2.1.0/30 -d 10.2.1.2 -j SNAT --to-source 10.2.1.1

        #post-up   iptables -t nat -A POSTROUTING -o enp0s31f6 -s 10.2.1.2 -j SNAT --to-source 94.x.x.B
        #post-down iptables -t nat -D POSTROUTING -o enp0s31f6 -s 10.2.1.2 -j SNAT --to-source 94.x.x.B
        #post-up   iptables -t nat -A PREROUTING -i enp0s31f6 -d 94.x.x.B -j DNAT --to-destination 10.2.1.2
        #post-down iptables -t nat -D PREROUTING -i enp0s31f6 -d 94.x.x.B -j DNAT --to-destination 10.2.1.2
        post-up   iptables -A FORWARD -s 94.x.x.B -j ACCEPT
        post-down iptables -D FORWARD -s 94.x.x.B -j ACCEPT
        post-up   iptables -A FORWARD -d 10.2.1.2 -j ACCEPT
        post-down iptables -D FORWARD -d 10.2.1.2 -j ACCEPT

        #post-up   iptables -P FORWARD ACCEPT
        #post-up   iptables -P OUTPUT ACCEPT
        #post-up   iptables -P INPUT ACCEPT
        #post-up   iptables -t nat -A POSTROUTING -o enp0s31f6 -s 10.2.1.0/30 ! -d 10.2.1.0/30 -j SNAT --to-source 94.x.x.B
        #post-down iptables -t nat -D POSTROUTING -o enp0s31f6 -s 10.2.1.0/30 ! -d 10.2.1.0/30 -j SNAT --to-source 94.x.x.B
        #post-up   iptables -A FORWARD -i enp0s31f6 -o vmbr0 --syn -m conntrack --ctstate NEW -j ACCEPT
        #post-down iptables -D FORWARD -i enp0s31f6 -o vmbr0 --syn -m conntrack --ctstate NEW -j ACCEPT
        #post-up   iptables -A FORWARD -i enp0s31f6 -o vmbr0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
        #post-down iptables -D FORWARD -i enp0s31f6 -o vmbr0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
        #post-up   iptables -A FORWARD -i vmbr0 -o enp0s31f6 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
        #post-down iptables -D FORWARD -i vmbr0 -o enp0s31f6 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
        #post-up   iptables -A FORWARD -d 10.2.1.2/32 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
        #post-down iptables -D FORWARD -d 10.2.1.2/32 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
        #post-up   iptables -A FORWARD -s 10.2.1.0/30 -j ACCEPT
        #post-down iptables -D FORWARD -s 10.2.1.0/30 -j ACCEPT
        #post-up   iptables -A FORWARD -d 10.2.1.0/30 -j ACCEPT
        #post-down iptables -D FORWARD -d 10.2.1.0/30 -j ACCEPT
        #post-up   iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT
        #post-down iptables -D INPUT -m state --state ESTABLISHED -j ACCEPT
        #post-up   iptables -t nat -A PREROUTING -i enp0s31f6 -d 94.x.x.B -j DNAT --to-destination 10.2.1.2
        #post-down iptables -t nat -D PREROUTING -i enp0s31f6 -d 94.x.x.B -j DNAT --to-destination 10.2.1.2
        #post-up   iptables -t nat -A PREROUTING -d 94.x.x.B -j DNAT --to-destination 10.2.1.2
        #post-down iptables -t nat -D PREROUTING -d 94.x.x.B -j DNAT --to-destination 10.2.1.2
        #post-up   iptables -t nat -A POSTROUTING -o vmbr0 -d 10.2.1.2 -j SNAT --to-source 10.2.1.1
        #post-down iptables -t nat -D POSTROUTING -o vmbr0 -d 10.2.1.2 -j SNAT --to-source 10.2.1.1

        #post-up   iptables -t nat -A PREROUTING -d 94.x.x.B -j DNAT --to 10.2.1.2
        #post-down iptables -t nat -D PREROUTING -d 94.x.x.B -j DNAT --to 10.2.1.2
        #post-up   iptables -t nat -A POSTROUTING -s 10.2.1.2 -j SNAT --to 94.x.x.B
        #post-down iptables -t nat -D POSTROUTING -s 10.2.1.2 -j SNAT --to 94.x.x.B

        #post-up   iptables -t nat -A POSTROUTING -s 10.2.1.0/30 -d 10.2.1.2 -j LOG --log-level info
        #post-up   iptables -t nat -A POSTROUTING -s 10.2.1.0/30 -d 10.2.1.2 -j SNAT --to-source 10.2.1.1
        #post-down iptables -t nat -D POSTROUTING -s 10.2.1.0/30 -d 10.2.1.2 -j SNAT --to-source 10.2.1.1
        #post-up   iptables -t nat -A POSTROUTING -o enp0s31f6 -s 10.2.1.0/30 -j LOG --log-level info
        #post-up   iptables -t nat -A POSTROUTING -o enp0s31f6 -s 10.2.1.0/30 -j SNAT --to-source 94.x.x.B
        #post-down iptables -t nat -D POSTROUTING -o enp0s31f6 -s 10.2.1.0/30 -j SNAT --to-source 94.x.x.B
        #post-up   iptables -t nat -A POSTROUTING -s 10.2.1.0/30 -j LOG --log-level info
        #post-up   iptables -t nat -A POSTROUTING -s 10.2.1.0/30 -j MASQUERADE
        #post-down iptables -t nat -D POSTROUTING -s 10.2.1.0/30 -j MASQUERADE
        post-up    iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
        post-down  iptables -t raw -D PREROUTING  -i fwbr+ -j CT --zone 1
        #post-up    iptables -I FORWARD -s 94.x.x.B -j ACCEPT
        #post-down  iptables -D FORWARD -s 94.x.x.B -j ACCEPT
        #post-up    iptables -I FORWARD -d 10.2.1.0/30 -j ACCEPT
        #post-down  iptables -D FORWARD -d 10.2.1.0/30 -j ACCEPT
Improve this question
4
  • who/where is the hoster? most hoster denies that different macs access the network like netcup do. since they only allow the primary Mac to access it is needed to enable arp proxy and route/nat the access. Even more if the ip is dedicated to one vm, why don't u assign it directly and route it?
    – djdomi
    Jan 23, 2023 at 17:49
  • "docker": as docker itself does a lot of network settings, you should first try your setup on a system where Docker is not installed at all (or was disabled and then the system rebooted).
    – A.B
    Jan 23, 2023 at 20:22
  • @djdomi Running on Hetzner. Yes, I read that in proxmox routed section (pve.proxmox.com/wiki/Network_Configuration): "Most hosting providers do not support the above setup. For security reasons, they disable networking as soon as they detect multiple MAC addresses on a single interface." That's why I'm using Nat. To your question, why not assigning directly the IP: I don't know how I would do that with my current NAT setup.
    – Leo
    Jan 24, 2023 at 2:09
  • @A.B I don't think docker is causing issues here. As I said, accessing the web interface via the local IP was possible, that shows me that the containers are able to communicate wich each other as they should. Also I had the above mentioned checks, as well as the UI, working when it was even behind my virtualised opnsense/pfsense. The only issue I had there was the microphone and video session was throwing an error. Everything else seemed to work. I think it's the iptables NAT reflection part which kills me here.
    – Leo
    Jan 24, 2023 at 2:16

2 Answers 2

Reset to default
1

Couldn't make BigBlueButton work with 1:1 NAT + Reflection, so I gave up on that approach. I still found a solution using an approach @djdomi mentioned: "why don't u assign it directly and route it?" which I documented here: Assigned second public IP to VM from outside not reachable

Improve this answer
1
  • the reason why you need to route it is, because its beyond the bridge and assigned to the VM directly. else the bridge has the IP and share it with the vms.
    – djdomi
    May 29, 2023 at 10:43
1

Since you use Hetzner which has similar Requirements as on Netcup as I use.

As you did not post the complete config, I will do that for you

#default interaces
auto lo
iface lo inet loopback
iface lo inet6 loopback

#ens3 could be other named 
auto ens3
#8.8.8.8 = pub ip, 1.1.1.1 = gateway IP and PtP
iface ens3 inet static
    address 8.8.8.8/32
    gateway 1.1.1.1
    pointopoint 1.1.1.1
    #Init all Pre-Requirements for vmbr0
    post-up echo 1 > /proc/sys/net/ipv4/conf/ens3/proxy_arp
    post-up echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
    #Restore persistent the IPTables
    pre-up iptables-restore < /etc/iptables.rules
    #dump that for reboot
    pre-down iptables-save > /etc/iptables.rules
    


auto vmbr0 
iface vmbr0 inet static
    address     10.0.0.1/8
        bridge_ports    none
        bridge_stp  off
        bridge_fd   0
        post-up     echo 1 > /proc/sys/net/ipv4/ip_forward
    #nating enable on vmbr0 up and wise versa when disabled
    post-up         iptables -t nat -A POSTROUTING -s '10.0.0.0/8' -o vmbr0 -j MASQUERADE
    post-down       iptables -t nat -D POSTROUTING -s '10.0.0.0/8' -o vmbr0 -j MASQUERADE
    #Route 2 ips direct over vmbr0 to the VM's where 2.2.2.* is the public ip
        post-up         route add 2.2.2.2 dev vmbr0
        post-up         route add 2.2.2.3 dev vmbr0
    #Remove that for reboot, and cleanup
        post-down       route del 2.2.2.2 dev vmbr0
        post-down       route del 2.2.2.3 dev vmbr0

Basically, what this configuration do, is that it uses the main MAC from the Host, instead of publishing all macs to the network.

Why is that important on some Hoster?

Due to Limitations on Security site or ARP-Tables, most user deny accessing the network else as the MAC from the known server. Moreover, it could happen, Specially for Proxmox, that MAC-Adresses could exist multiple times specifically in case of a Data-Center, where thousands of costumer do the same thing.

What does this config do other?

Short Answer:

Use ENS3 Mac and proxy the rest of the VM's by using NAT I am using the 10/8 as I am lazy, and want to use subnets to identify the service beyond Any VM will need to put 10.0.0.1 as gateway to access the internet, regardless if they have been assigned a Public IP or both.

Long Answer:

we need to dig deeply into this:

ens3

At First, ENS3 in this case is for Netcup the default NIC. This has to be replaced by you with your current setup. The Gateway and the PoinToPoint have to be the same. The Reason is, that it redirects any unknown traffic to the gateway and mostly removes some head cache wired bugs, that I cannot explain, but happened in the past for having no connectivity.

At Second, on Post-UP (Enabling the Device) it already enables the device as a proxy for ARP. As also Forward the IPv6 traffic. (needs a separate configuration!)

And Also it imports the existing iptables, in case, it was already a shutdown/disabling of the device.

vmbr0

So Mostly, the people will see that there are no bridge-ports has been used. Yes, the reason for that is, that we will use a "routed" instead of a "bridged" setup. Due to the ARP proxying and NAT, we have no issues. But will solve the Limitations.

Basically, we enable ip_forward every time in case it's getting somehow disabled. Then we enable on the device the NAT-Rules on the fly and disable them, when the bridge goes down. That is important, due that sometimes a rare connectivity-issue can happen, where then only a reboot helps. The same is for the Public-IP's for the VMs → And Mostly you think "WTH - I did not change anything?? Why does my VM's not work?" → Moreover, no you assign the route to the Bridge, not the Virtual Interface. The Reason is, the Virtual Interface is after the bridge and already have the information about (internet ⇒ ens3 ⇒ vmbr0 ⇒ virtual iface)

Conclusion

There is no "perfect" configuration for any hoster. It Mostly depends on the requirements and limitations. In my point of view, this configuration works for years now.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .