How can I fix this wrong/incomplete Xen routing setup?

Question

I've got some Xen hosts configured in bridged mode, whose VMs use public or private IPs:

Host #1

• dom0: 3.3.3.71
• dom1: 10.0.2.10
• dom2: 3.3.3.85
• dom3: 10.0.2.11

Host #2

• dom0: 3.3.3.72
• dom1: 3.3.3.86
• dom2: 10.0.2.12
• dom3: 10.0.2.13

Host #3

• dom0: 3.3.3.73
• dom1: 10.0.2.14
• dom2: 10.0.2.15
• dom3: 3.3.3.87

VMs with public IPs (3.3.3.*) are reachable from the Internet. VMs with private IPs are reachable from within each other, and this is a desired behaviour. However, they can't reach the Internet because there's no 10.0.2.1 gateway in the local subnet. How can I fix this?

One solution would be to configure one server with a 10.0.2.1 interface to act as a NAT. However I don't want any SPOF and I'd like each host to do its own NAT using the public IP address assigned to the dom0.

I believe this is just a matter of configuring iptables in the dom0's. If this is correct, how?

Answer 1

I found the following solution. I'm still open to good alternatives and hints.

On each Xen host:

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -m physdev --physdev-in vif5.0 -o eth0 \
    -j SNAT --to-source 3.3.3.71

On each domU having a private IP (10.0.2.*):

route add -net 3.3.3.0 netmask 255.255.255.0 dev eth0
route add default gw 3.3.3.71
route del default gw 10.0.2.1

Answer 2

Is there any reason why you don't connect the DomUs that need access to the internet to the internet-bridges?

If you don`t want that - connect one of your DomUs that are connected to the internet to the "private" bridges and install a proxy on then (squid). Contact that proxy...

I don*t think it is a good idea to connect a Dom0 to the internet - but you will have your reasons...