Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm having trouble getting floating IPs to work properly on my OpenStack setup. It looks like the controller node (running nova-network) isn't forwarding the traffic to the instances.

I've got a Folsom deployment with FlatDHCP, not running multi-host, running on Ubuntu 12.04.

As an example, here's a running instance with a fixed IP of 10.40.0.2 and a floating IP of 10.20.0.3:

$ nova list
+-------+---------+--------+------------------------------+
| ID    | Name    | Status | Networks                     |
+-------+---------+--------+------------------------------+
| 3d292 | quantal | ACTIVE | private=10.40.0.2, 10.20.0.3 |
+-------+---------+--------+------------------------------+

If I'm logged into the controller, I can ping and ssh to the VM instance from either of the IPs. However, I cannot ping or ssh to the instance from an external machine.

If I try to ping from a machine on a different network (192.168.3.5), I can see the packets reach the controller on its public interface (eth3), but those packets aren't getting forwarded to the bridge (br100) that's connected to the private interface:

# tcpdump -i any icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
20:33:12.908188 IP 192.168.3.5 > 10.20.0.3: ICMP echo request, id 58378, seq 0, length 64
20:33:13.910759 IP 192.168.3.5 > 10.20.0.3: ICMP echo request, id 58378, seq 1, length 64
20:33:14.910591 IP 192.168.3.5 > 10.20.0.3: ICMP echo request, id 58378, seq 2, length 64

I've configured nova-network to use the "no-op" firewall driver, so there shouldn't be any security group issues here:

firewall_driver=nova.virt.firewall.NoopFirewallDriver

I've confirmed that I have IP forwarding enabled:

$ cat /proc/sys/net/ipv4/ip_forward
1

I'm not too familiar with iptables, but looking at the rules (see https://gist.github.com/lorin/5209761), I can't see anything obviously wrong.

I assume this rule is in effect when connecting from the controller to the VM:

-A nova-network-OUTPUT -d 10.20.0.3/32 -j DNAT --to-destination 10.40.0.2

And that it hits these rules when doing forward

-A nova-network-PREROUTING -d 10.20.0.3/32 -j DNAT --to-destination 10.40.0.2
-A nova-network-FORWARD -o br100 -j ACCEPT

At this point, I'm at a loss as to why it isn't doing the NAT properly and forwarding the packets.

The interfaces are:

  • eth0: (not involved here)
  • eth1: connects to the internal VM-only network, br100 is bridged to it. Packets should ultimately go out that interface
  • eth2: (not involved here)
  • eth3: public-facing interface. It has the IP of the controller (10.20.0.2), as well as the floating IP of the instance (10.20.0.3).

Below is the output of "ip a", with MACs obscured.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 169.254.169.254/32 scope link lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether aa:bb:cc:dd:ee:ff brd ff:ff:ff:ff:ff:ff
    inet 10.30.0.131/24 brd 10.30.0.255 scope global eth0
    inet6 ef23::2345:3aff:fe86:50c8/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br100 state UP qlen 1000
    link/ether 11:22:33:44:55:66 brd ff:ff:ff:ff:ff:ff
    inet6 ef23::2345:3aff:fe86:50c9/64 scope link
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 77:88:99:00:11:22 brd ff:ff:ff:ff:ff:ff
    inet 10.10.0.131/24 brd 10.10.0.255 scope global eth2
    inet6 ef23::2345:3aff:fe68:b4b6/64 scope link
       valid_lft forever preferred_lft forever
5: eth3: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 33:44:55:66:77:88 brd ff:ff:ff:ff:ff:ff
    inet 10.20.0.2/24 brd 10.20.0.255 scope global eth3
    inet 10.20.0.3/32 scope global eth3
    inet6 ef23::2345:3aff:fe68:b4b7/64 scope link
       valid_lft forever preferred_lft forever
6: br100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 54:78:1a:86:50:c9 brd ff:ff:ff:ff:ff:ff
    inet 10.40.0.1/16 brd 10.40.255.255 scope global br100
    inet6 ef23::2345:3aff:fed1:cd55/64 scope link
       valid_lft forever preferred_lft forever

Routing table:

~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.10.0.1       0.0.0.0         UG    100    0        0 eth2
10.10.0.0       0.0.0.0         255.255.255.0   U     0      0        0 eth2
10.20.0.0       0.0.0.0         255.255.255.0   U     0      0        0 eth3
10.30.0.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0
10.40.0.0       0.0.0.0         255.255.0.0     U     0      0        0 br100

Bridge setup:

# brctl show
bridge name bridge id       STP enabled interfaces
br100       8000.54781a8650c9   no      eth1

Here are some networking lines from /etc/nova/nova.conf

network_manager=nova.network.manager.FlatDHCPManager
flat_interface=eth1
flat_network_bridge=br100
public_interface=eth3
firewall_driver=nova.virt.firewall.NoopFirewallDriver
share|improve this question
add comment

1 Answer

up vote 2 down vote accepted

You say under "The interfaces are" that eth2 isn't involved but from the route output it looks like the gateway is set up for eth2.

I think you would want it set up on eth3. I'd check that /etc/network/interfaces is set up to reflect that.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.