Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We work with devices that are flashed (factory) using TFTP on a usb network connection.

The server has a fixed address and the device a fixed address. When it starts, it connects to download the firmware.

In the current setup, only one device can work at the same time. But I'd like to make it possible to flash as many devices as we can plug (because we can have some massive flashing requirements).

To bypass the routing issue, I made a xinetd version that sets a setsockopt to SO_BIND_DEVICE.

But what I didn't except is that Linux is unable to handle the ARP requests on both interfaces at the same time.

if I do a "ping -I usb0" and "ping -I usb1" at the same time, it will work on one interface:

ARP, Request who-has sk tell, length 28
ARP, Reply is-at 7a:0f:66:7c:fc:2c (oui Unknown), length 28
IP > ICMP echo request, id 21807, seq 1, length 64
IP > ICMP echo reply, id 21807, seq 1, length 64

But on the other it won't:

IP > ICMP echo request, id 31071, seq 1, length 64
ARP, Request who-has tell, length 28
ARP, Request who-has tell, length 28
IP > ICMP echo request, id 31071, seq 2, length 64
IP > ICMP echo request, id 31077, seq 1, length 64
ARP, Request who-has tell, length 28

The server doesn't seem to answer to the ARP request.

This is how the connection from the device is handled on the server with a /etc/network/if-up.d/000-first script :

ifconfig $IFACE up
ifconfig $IFACE


# this is the modified xinetd version to bind on one address
kill -9 `cat $PID`
xinetd -pidfile $PID -interface $IFACE

# I tried this to force the handling of ARP table per interface, but it doesn't change anything:
# /usr/sbin/arpd -b /tmp/$IFACE.db -a 3 -k $IFACE 

Here is the modified xinetd version:

share|improve this question
I am kinda wondering if you could do some magic with ebtables NAT rules. I know you theoretically can mess aground with arp, and mangle the IP addresses. Perhaps with the right magic you could get the tftp server to believe the remote hosts had different source addresses. – Zoredache Sep 7 '13 at 8:33
The core logic of the device updating process is to contact the server. The ARP request is for, I can't change the IP the device asks for on the ARP layer (as far as I know). I'm a little bit frustrated by this situation. Basically I would just need each interface to be able to reply to the ARP request on the same interface. Even arpd doesn't work. Maybe I'm missing something here... – Florent Sep 8 '13 at 2:00
up vote 2 down vote accepted

First, if you haven't tried proxy arp by itself, you should try proxy arp.

echo 1 > /proc/sys/net/ipv4/conf/all/proxy_arp

Option 1: some connmark/policy route mangling

If proxy arp alone didn't work, I can't guarantee this will work, but it's worth a shot, I think. What I'm about to describe is how to set up policy routing to bind responses based on input interface using conntrack. I will describe how to set it up for two interfaces, and it should be easy enough to expand to any number. On the other hand, instead of working, it may simply get mightily confused and not do a damn thing (probably because the kernel arp table doesn't know it should cache ip/mac pairs per device). If that happens, try the second, more ugly method.

First, set up some connmark mangle rules:

iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m mark ! --mark 0 -j ACCEPT

In /etc/iproute2/rt_tables add two tables, 100 and 101, label them usb0 and usb1

100    usb0
101    usb1

For each of these tables add the following (replace <N> with the appropriate number):

ip route add dev usb<N> table usb<N>
ip rule add fw <N> table usb<N>
iptables -t mangle -A PREROUTING -i usb<N> -j MARK --set-mark <N>

I'm not certain, but you may need to set up a dummy interface like so for your tftp daemon to listen on:

modprobe dummy
ifconfig dummy0 up
echo 1 > /proc/sys/net/ipv4/conf/all/proxy_arp
sysctl net.ipv4.ip_forward=1
iptables -I FORWARD -s -j ACCEPT
iptables -I FORWARD -d -j ACCEPT

Now test with two devices. If it works, excellent, otherwise try the next method:

Option 2: kvm container per usb net device

Option 2 is uglier, involving some bridging and qemu/kvm. virt-manager is probably the easiest way to create these things.

Create tap devices and bridges, one per usb interface.

tunctl -t tap<N>
brctl addbr usb<N>br
brctl addif usb<N>br tap<N>
brctl addif usb<N>br usb<N>  # this may need to be done each time the usb device is connected/disconnected.

Create a kvm image file or boot CD that is designed to be read only and hosts your tftp server and image... or omit read-only and make one image file per vm (using snapshots for this would be better, but that's more out of scope than this answer is getting).

Run a kvm using the tap interface and your image file and test usb network connectivity.


If connmark/policy route worked, it's much easier to synchronously maintain the tftp repository. With KVM, you may need to have a repository per VM image (unless you pass a read-only directory through to each VM, which is doable). Connmark/policy route is a bit more finicky, but it also only requires one tftp server, though if your ports aren't random enough, conntrack could get confused and overwrite if there is a port overlap (or it may not). On the other hand, bridging to kvms requires much more memory; an entire VM per attached USB device, but it's much more likely to work every time because the VM has an entire isolated network stack to work with while the host kernel just has to pass packets back and forth over a bridge, especially true if bridge filtering is off.

Hopefully one of these two work for you.

share|improve this answer
This is a side project and I still haven't found the time to test option 1. ProxyARP doesn't work. Option 2 is not possible because the goal is to allow anyone to update devices with a small device like a raspberry. – Florent Sep 16 '13 at 11:59
@Florent If you're designing the device-being-configured, you could have it probe for dhcp first and if it returned a tftp image, load from that address and otherwise fall back to a default address. That addresses both your needs for mass configuration and sane end-user defaults. – Andrew Domaszek Sep 16 '13 at 14:34
Yes definitely. This is something that is planned but the current bootloader doesn't support it. And I'm not working on this part of the software. – Florent Sep 30 '13 at 20:37

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.