I have sporadic routing problems in my setup which I suspect to be an issue with the routing and two network interfaces.

  • I'm running two maschines with XenServer
  • They both have a public NIC with an own IP for internet traffic
  • They both have a private NIC with a Cross-Over-Cable for local traffic

The VMs both also have two (virtual) NICs, eth0 with their own public IP and eth1 for the local traffic.

Occasionally I can reach the servers from the outside but they are not able to ping/reach each other locally.


          Public IP               local IP
xh01      aa.bb.10.214  
xh02      aa.bb.10.215  
lb01      aa.bb.10.242  
lb02      aa.bb.10.241  
be01      aa.bb.10.239  
be02      aa.bb.10.240  

Example routes on lb01:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface         aa.bb.10.215         UG    100    0        0 eth0
aa.bb.10.192    aa.bb.10.193 UG    0      0        0 eth0
aa.bb.10.193 UH    0      0        0 eth0   U     0      0        0 eth1

Now, sometimes for example I can't ping lb01 from be01.

With 'tcpdump' on lb01 I can see that the ICMP echo request reaches lb01 but I can't see an answer. Also, if I issue a tracepath on one of the hosts to the other, the ping suddenly gets a reply.

My suspicion is, that since the maschines can reach each other via both interfaces, outgoing traffic of a connection is not routet to the same interface as the incoming. I guess that would mean that I have to set up different routing tables?

Could someone with a deeper knowledge in the whole routing stuff give me a few hints on how fix this?

  • Which IP address are you pinging, and what firewall rules do you have set up? Sep 16 '13 at 7:49
  • You only need one routing table. Each host should have an entry in their routing table like the last one above (192.168.1.x reachable via eth1 - which should be create automatically). This shows that it knows it can reach the 192.168.1 network via eth1. The fact that they can also see each other via their public addresses on eth0 shouldn't affect traffic using the 192.168.1 network.
    – USD Matt
    Sep 16 '13 at 7:51
  • @FalconMomot: I'm pinging the public address. Local seems to work always. No firewall rules since iptables is running on the XenServer hosts.
    – Bannane
    Sep 16 '13 at 8:19

There isn't a routing problem visible from the routing table on the host you have posted. However, I notice you are using some very awkward behaviour whereby you set two gateways that are in subnets you don't have an address in, one of which you don't have a local route for, which makes me wonder whether you have hit some undefined behaviour. I would be curious to see the contents of the ARP tables on the hosts while the ping isn't going through.

I think I see what you are trying to do here - this type of thing is nonstandard. It seems somewhat purposeless to try to force traffic through a gateway on one network but only do it for hosts which are otherwise directly connected on some other subnet, though if you were to want to send traffic through a router the correct solution is to provision a /30 subnet for each host.

Failing that, it's possible that the gateway is doing some kind of stateful processing (such as m_conntrack, if it is running linux) and is severely unprovisioned, or you might be having packet loss. Alternatively, the Xen server might be taking time to figure out where the packets go, particularly if you are using openvswitch which has a lot of surprising stateful behaviour. It might also be requiring that some outbound traffic be sent before inbound traffic can be received, though this would be unusual.

It is certainly not the case that traffic to your non-RFC1918 network will be transmitted over eth1, if the routing table looks like it does. There is no route that would do this. Also, bear in mind that each host has one routing table in which routes are associated with interfaces. Any packet which is needing routing (basically, an outbound one) traverses this same table and follows the same rules, matching the most specific route, using the metric to break ties.

  • Thanks for your comprehensive description! So, since xh01 (.215) is connected to the subnet via the gateway setting it as the gateway for the other hosts should solve the problem? Would that be more "standandard conform"?
    – Bannane
    Sep 16 '13 at 9:21
  • Hmm? No, I think you might be confused - the next hop in a route can never be through a gateway. It must be on the local segment, and generally should be in a subnet on which you have an address and for which you have a local route (local routes are those which do not have gateways). This is because the routing table lookup is used to find the next hop; it cannot be used to specify the path taken after that. Sep 16 '13 at 9:33
  • But that would be the case with .215 as the gateway, wouldn't it? Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface aa.bb.10.215 UG 100 0 0 eth0 aa.bb.4.170 aa.bb.10.193 UGH 0 0 0 eth0 aa.bb.10.192 aa.bb.10.215 UG 0 0 0 eth0 aa.bb.10.192 U 0 0 0 eth0 U 0 0 0 eth1
    – Bannane
    Sep 16 '13 at 9:42
  • I changed the gateways and it seems to work now. Apparently the router drops all packets wich are not comming from the physical hosts MAC. Just makes me wonder why it worked occasionally?
    – Bannane
    Sep 16 '13 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.