4

I have a dedicated hosting with hetzner. Additionally i have bought a 6IP subnet.

My main IP is: 148.111.111.200 My main subnet is: 255.255.255.224

My additional IPs are 148.222.222.1 to 148.222.222.6.

My usage scenario is as follows: Several instances will have public facing IPs (webservers, etc) All instances will have a second nic setup for internal LAN, so I can have mysql server, memcached, etc on private network locked from outside.

Webserver will be online on 148.222.222.1 and will have a second NIC with ip 10.10.10.10

Currently, i have setup the internal lan. All the instances can connect and ping one another via internal IP (10.10.10.X), but my webserver cannot reach the internet.

I cannot use bridged mode, since hetzner does not allow multiple MACs on same external ip, so I have to use routing mode. Here is my /etc/network/interfaces file for the host:

# network interface settings
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static

    address  148.111.111.200
    netmask  255.255.255.255
    pointopoint 148.111.111.193
    gateway  148.111.111.193
    broadcast  148.111.111.223
    post-up echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
#commentedout
    #up route add -net 148.111.111.192 netmask 255.255.255.224 gw 148.111.111.193 eth0


    # default route to access subnet

auto vmbr0
iface vmbr0 inet static

    address  148.111.111.200
    netmask  255.255.255.255
    bridge_ports none
    bridge_stp off
    bridge_fd 0

    up ip route add 148.222.222.1/32 dev vmbr0
    up ip route add 148.222.222.2/32 dev vmbr0

auto vmbr1
iface vmbr1 inet static

    address 10.10.10.1
    netmask 255.255.255.0
    bridge_ports none
    bridge_stp off
    bridge_fd 0

    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eth0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eth0 -j MASQUERADE

auto vmbr2
iface vmbr2 inet static

    address  148.222.222.1
    netmask  255.255.255.248
    bridge_ports none
    bridge_stp off
    bridge_fd 0

And here is my kvm interfaces file:

auto eth0
iface eth0 inet static
    address 148.222.222.1
    netmask 255.255.255.255
    pointopoint 148.111.111.200
    gateway 148.111.111.200
    dns-nameservers 8.8.8.8 8.8.4.4

auto eth1
    address 10.10.10.12
    netmask 255.255.255.0
    network 10.10.10.0
    broadcast 10.10.10.255

Currently, KVM instances can ping oneanother, but nothing else works Host can ping instances, but nothing else And i cannot get internet access on my instances.

What do I need to change in my configuration in order for this to work.

P.S. NIC type is set to virtio in proxmox

3 Answers 3

3

I solved the problem by using the following config:

auto eth0
iface eth0 inet static
    address  148.111.111.200
    netmask  255.255.255.255
    pointopoint 148.111.111.193
    gateway  148.111.111.193
    broadcast  148.111.111.193

# default route to access subnet

auto vmbr0
iface vmbr0 inet static
    address  148.111.111.200
    netmask  255.255.255.255
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0

    #subnet
    up ip route add 148.222.222.0/32 dev vmbr0
    up ip route add 148.222.222.1/32 dev vmbr0
    up ip route add 148.222.222.2/32 dev vmbr0
    up ip route add 148.222.222.3/32 dev vmbr0
    up ip route add 148.222.222.4/32 dev vmbr0
    up ip route add 148.222.222.5/32 dev vmbr0
    up ip route add 148.222.222.6/32 dev vmbr0
    up ip route add 148.222.222.7/32 dev vmbr0

auto vmbr1
iface vmbr1 inet static
    address 10.10.10.1
    netmask 255.255.255.0
    bridge_ports none
    bridge_stp off
    bridge_fd 0

    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eth0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eth0 -j MASQUERADE

On your client machines you need to define one ore two network interfaces

  • One for VMBR0 (public facing IP)
  • One for VMBR1 (private IP - 10.10.10.X)

Sample config for ubuntu /etc/network/interfaces

#vmbr0
auto eth0
iface eth0 inet static
  address 148.222.222.1
  netmask 255.255.255.255
  pointopoint 148.111.111.200
  gateway 148.111.111.200 #public IP for the proxmox node
  dns-nameservers 8.8.8.8 8.8.4.4

#vmbr1
auto eth1
iface eth1 inet static
  address 10.10.10.20
  netmask 255.255.255.0
  network 10.10.10.0
  broadcast 10.10.10.255
0

Looking at your config for vmbr0, it seems like you are pointing routing to devices that the interface cannot reach (ie, to devices which the netmask puts outside the interface's network reach).

Without digging too much into your config, I would make the following suggestions to simplify your setup now and for the future:

  • Create a VM that will act as your UTM (perhaps running pfsense, vyatta, astaro, zentyal, clearos, untangle, etc). This will ensure you worry about connectivity only on one or two nodes that play gateway (and redundant gateway) roles. Then all your routing and access logic will happen within this UTM VM. I have personally successfully done this with both pfsense and zentyal
  • Proxmox VE 3.2 now works with openvswitch, try and deploy atop that. This will give you ease of administrating your network stacks, without delving too much into the innards of ebtables and iptables rules.

Also, you are configuring networking on both the eth{0,1} interfaces and the bridges atop of them. I would typically suggest taking off the configs from the eth{0,1} interfaces and configuring networking only on the bridge interfaces.

How about the following, which takes interface config details away from the host and also leaves control for routing between networks to your UTM VM .. :


auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto vmbr0
iface vmbr0 inet manual
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0

auto vmbr1
iface vmbr1 inet manual
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0

auto vmbr2
iface vmbr2 inet static
    address  10.10.10.12
    netmask  255.255.255.0
    gateway  10.10.10.1
    bridge_ports eth1
    bridge_stp off
    bridge_fd 0

Obviously if connectivity to the host is an issue, you will have to modify vmbr0 or vmbr1 and add an IP for the host .. The fundamental idea is to not burden the host with inter-VM network routing and have one dedicated VM take care of those details

With this setup, your UTM VM becomes mighty powerful, and you will even have the host forwarding routing to it for the externally accessible IP addresses. In that scenario, I would recommend running proxmox in a cluster(even just 2-node cluster), and enabling HA for the UTM VM.

0

also you need this at the top

auto lo

iface lo inet loopback

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address  148.111.111.200
    netmask  255.255.255.255
    pointopoint 148.111.111.193
    gateway  148.111.111.193
    broadcast  148.111.111.193

# default route to access subnet

auto vmbr0
iface vmbr0 inet static
    address  148.111.111.200
    netmask  255.255.255.255
    bridge_ports none
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0

    #subnet
    up ip route add 148.222.222.0/32 dev vmbr0
    up ip route add 148.222.222.1/32 dev vmbr0
    up ip route add 148.222.222.2/32 dev vmbr0
    up ip route add 148.222.222.3/32 dev vmbr0
    up ip route add 148.222.222.4/32 dev vmbr0
    up ip route add 148.222.222.5/32 dev vmbr0
    up ip route add 148.222.222.6/32 dev vmbr0
    up ip route add 148.222.222.7/32 dev vmbr0

auto vmbr1
iface vmbr1 inet static
    address 10.10.10.1
    netmask 255.255.255.0
    bridge_ports none
    bridge_stp off
    bridge_fd 0

    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eth0 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eth0 -j MASQUERADE

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .