Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Trying to troubleshoot a problem where DHCP request packets aren't getting into a KVM virtual machine. The DHCP server and the hypervisor are on separate machines, separated by a swich.

br100 <===> eth1  <==== Cisco Nexus 3000 ===> eth1 <===> br100 <===> vnet0 <===> [eth0 inside of VM]

If I tcpdump on vnet0, I can see the DHCP requests getting there, but they aren't making their way into the instance.

However, if I sniff the packets with scapy, and then retransmit them, then they get in!

In other words, if I run this script on the compute node that runs KVM, then the guest will see the DHCP packets and it wil recognize its IP address.

from scapy.all import *

def lfilter(x):
    return x.haslayer(BOOTP) and x.sport==67 and x.dport==68 and x.src=='54:78:1a:86:50:c9'

conf.iface = 'vnet0'

for i in range(2):
    x = sniff(iface='vnet0', lfilter=lfilter, count=1)
    sendp(x, iface="vnet0")

What could be the difference between the packets that are travelling from the DHCP server and the same packets that are retransmitted? Could the switch or the NICs somehow be adding something to the packets that scapy is stripping out? What else could it be?

Edit:

IP forwarding is enabled on the compute host that is running the virtual machine:

# cat /proc/sys/net/ipv4/ip_forward
1

Here is the output of iptables-save on the compute host that is running the virtual machine.

# Generated by iptables-save v1.4.12 on Fri Apr  5 20:29:22 2013
*nat
:PREROUTING ACCEPT [14519:4031039]
:INPUT ACCEPT [10964:3406988]
:OUTPUT ACCEPT [7710:575151]
:POSTROUTING ACCEPT [10972:1135076]
:nova-compute-OUTPUT - [0:0]
:nova-compute-POSTROUTING - [0:0]
:nova-compute-PREROUTING - [0:0]
:nova-compute-float-snat - [0:0]
:nova-compute-snat - [0:0]
:nova-postrouting-bottom - [0:0]
-A PREROUTING -j nova-compute-PREROUTING
-A OUTPUT -j nova-compute-OUTPUT
-A POSTROUTING -j nova-compute-POSTROUTING
-A POSTROUTING -j nova-postrouting-bottom
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A nova-compute-snat -j nova-compute-float-snat
-A nova-postrouting-bottom -j nova-compute-snat
COMMIT
# Completed on Fri Apr  5 20:29:22 2013
# Generated by iptables-save v1.4.12 on Fri Apr  5 20:29:22 2013
*mangle
:PREROUTING ACCEPT [87016:46950958]
:INPUT ACCEPT [86795:46911524]
:FORWARD ACCEPT [401:98474]
:OUTPUT ACCEPT [93709:37768035]
:POSTROUTING ACCEPT [94110:37866509]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
-A POSTROUTING -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Fri Apr  5 20:29:22 2013
# Generated by iptables-save v1.4.12 on Fri Apr  5 20:29:22 2013
*filter
:INPUT ACCEPT [3266325:18848348625]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [3572362:1405438279]
:nova-compute-FORWARD - [0:0]
:nova-compute-INPUT - [0:0]
:nova-compute-OUTPUT - [0:0]
:nova-compute-inst-35 - [0:0]
:nova-compute-local - [0:0]
:nova-compute-provider - [0:0]
:nova-compute-sg-fallback - [0:0]
:nova-filter-top - [0:0]
-A INPUT -j nova-compute-INPUT
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-compute-FORWARD
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-compute-OUTPUT
-A nova-compute-FORWARD -i br100 -j ACCEPT
-A nova-compute-FORWARD -o br100 -j ACCEPT
-A nova-compute-inst-35 -m state --state INVALID -j DROP
-A nova-compute-inst-35 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A nova-compute-inst-35 -j nova-compute-provider
-A nova-compute-inst-35 -s 10.40.0.1/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A nova-compute-inst-35 -s 10.40.0.0/16 -j ACCEPT
-A nova-compute-inst-35 -p tcp -m tcp --dport 22 -j ACCEPT
-A nova-compute-inst-35 -p icmp -j ACCEPT
-A nova-compute-inst-35 -j nova-compute-sg-fallback
-A nova-compute-local -d 10.40.0.4/32 -j nova-compute-inst-35
-A nova-compute-sg-fallback -j DROP
-A nova-filter-top -j nova-compute-local
COMMIT
# Completed on Fri Apr  5 20:29:22 2013
share|improve this question
    
tcpdump will help you identify the nature of the problem. Run this on both sides: tcpdump -n -e -v -i eth# '(ether MAC:OF:DHCP:SERVER or ether MAC:OF:DHCP:CLIENT) and (arp or port bootps or port bootpc)' If this isn't enough to show you the problem, share some of the output. (example of it working, example of it not) –  Andrew B Apr 5 '13 at 22:18
    
Are you retransmitting from the host? In that case, the difference would be the packets are not being forwarded--merely sent as output-- and so ipv4 forwarding does not need to be enabled and different netfilter chains are being traversed. If you manually assign an IP address does the guest work? –  Mark Wagner Apr 5 '13 at 23:27
    
@MarkWagner Yes, I'm retransmitting from the host. But do the ipv4 forwarding rules come into play when everything is happening at layer 2? And yes, the guest works if I manually assign an IP address. –  Lorin Hochstein Apr 5 '13 at 23:58
    
DHCP isn't layer 2. It uses IP so you need IP forwarding enabled and any netfilter rules allowing DHCP (UDP ports 67, 68, etc) traffic –  Mark Wagner Apr 6 '13 at 0:09
    
@MarkWagner I've confirmed that IP forwarding is enabled, and I've added the output of iptables-save. –  Lorin Hochstein Apr 6 '13 at 0:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.