Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

While debugging a problem with the networking configuration for a set of virtual machines running under KVM, I discovered a circumstance where the kernel in the guest VM has decided to stamp the outgoing Ethernet frame with a destination address that is in conflict with the address it would choose if it was respecting with the kernel IP route table.

So, in that example, I expected the outbound frame to be delivered to de:ad:be:3b:24:48 which corresponds to the host that owns the IP address 10.11.11.2 and that owns a route to 10.8.0.0/24.

What actually happened is that the kernel decided to stamp the frame with a destination of 00:10:db:ff:70:01 which sent the frame in the direction of 10.11.11.1 which doesn't know how to route to 10.8.0.0/24 and as a result the packet was dropped.

This decision was in contravention of the local routing table that clearly specified that the route to 10.8.0.0/24 was via 10.11.11.2. See the original problem report for details.

[The incorrect destination address was visible by running tcpdump in the guest VM that was sending the frame in the incorrect direction.].

In fact, by kludging the local arp table to make 10.11.11.1's apparent MAC address to be identical to 10.11.11.2's actual MAC address, I was able to get the frame flowing in the correct direction.

So my question is: what mechanism, in either the guest VM or the KVM host, could cause the guest kernel to ignore the local route table and send the packet in a frame towards the (incorrect) host at 10.11.11.1, even though 10.11.11.1 wasn't listed as a gateway for the destination network (10.8.0.0/24)?

Note: iptables was disabled in the guest at the time. I don't know if ebtables is enabled in the KVM host, but even if it was, how would this cause the kernel in the guest VM to want send the packet in the direction of 10.11.11.1?

One behaviour I did notice is that if I purge the ARP tables of the affected host and send a ping request to the affected host from the 10.8.0.0/24 network, it receives the request, then sends an arp broadcast for 10.11.11.1 immediately prior to sending the ping response in the direction of 10.11.11.1 instead of 10.11.11.2 and hence 10.8.0.0/24. What is causing it to try 10.11.11.1 which is not specified as a gateway?

share|improve this question

1 Answer 1

up vote 1 down vote accepted

Source or Policy based routing maybe? Linux can have several routing tables and choose which one to use based on several conditions. Check out the OpenVZ documentation on Source based routing

share|improve this answer
    
Thanks. As far as I know, in my particular case, there is no source based routing configured anywhere on the hosts (client, VPN server, LAN server) in my domain, so I don't think this explains the issue. –  jonseymour Mar 14 '12 at 9:25
    
Mmmm. Ok, so perhaps policy based routing is a factor here. When I run ip rule show, I get this: 0: from all lookup 255 32764: from 10.11.11.2 lookup route_eth1 32765: from 170.225.161.18 lookup route_eth0 32766: from all lookup main 32767: from all lookup default How do I inspect these tables, I wonder? –  jonseymour Mar 14 '12 at 9:29
    
ip route list table $id as per man ip(8) –  ptman Mar 14 '12 at 9:35
    
Ok, so this perhaps has provided the answer: /etc/sysconfig/network-scripts/route-eth0 has an entry of this kind: default table route_eth0 via 10.11.11.1 dev eth1 I suspect if I edit that to be 10.11.11.2 I'll get the desired behaviour without the arp hack. Let's see! –  jonseymour Mar 14 '12 at 9:36
    
Cool bananas, that did the trick - thanks! –  jonseymour Mar 14 '12 at 9:39

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.