I have a server with multiple interfaces and IP addresses on the same network. I would like for non-iSCSI traffic to exclusively use one of the interfaces, and for iSCSI traffic to exclusively use the rest of the interfaces. Limiting the iSCSI traffic to a subset of my interfaces is easy; I simply don't create entries in /var/lib/iscsi/ifaces/ for the interfaces I don't want it to use. However, I'm not sure what is a good approach to limiting the non-ISCSI traffic to one interface. As far as linux is concerned, the iSCSI and non-iSCSI interfaces are equally good routes to the network.

Here's an example configuration

The iSCSI storage has the IP addresses

The server has the following interfaces, addresses, and routes.

$ ip route list dev eth0  proto kernel  scope link  src dev eth1  proto kernel  scope link  src dev eth2  proto kernel  scope link  src dev eth3  proto kernel  scope link  src
default via dev eth0

The desired configuration is for eth3 to be used for non-ISCSI traffic and for eth1 and eth2 to be used for iSCSI traffic. However, non-iSCSI traffic currently goes out eth1.

$ ip route get to dev eth1  src

(some edits since original posting below)

With my current configuration, if eth1 and eth2 are fully saturated sending iSCSI traffic, my non-iSCSI traffic will compete with the iSCSI traffic on eth1 while eth3 sits idle.

How could I configure linux to prefer to send traffic to the local network using eth3 rather than eth1 or eth2?

I have already set net.ipv4.conf.all.arp_ignore to 1 and net.ipv4.conf.all.arp_announce to 2. This should prevent my IP addresses from floating between the interfaces, e.g. arp flux. I think I just need help with the routing.

(more edits)

Thanks to pfo I started looking at metrics. If I delete the routes and recreate them with the iSCSI interfaces having a higher metric than the non-iSCSI interface, things seem to work the way I want. iSCSI traffic still uses the dedicated interfaces without me setting up static routes to the iSCSI IP addresses. All other local traffic goes out eth3. Now I need to figure out the proper way to set the metrics automatically when the interfaces are brought up. This is on RHEL 5.5.

ip route delete to dev eth1
ip route delete to dev eth2
ip route delete to dev eth3
ip route add to dev eth1 src metric 1
ip route add to dev eth2 src metric 1
ip route add to dev eth3 src metric 0

(final update)

Assigning a different metric using the existing RHEL network scripts seems impossible, https://bugzilla.redhat.com/show_bug.cgi?id=498472

  • Are you fine with the trafic only using "its" interface each or do you require fallback if an interface fails?
    – Bittrance
    Apr 1 '11 at 20:46
  • 1
    Come to think of it, if all the interfaces are on the same net, what you really want is for traffic to be equally spread over all of them, no?
    – Bittrance
    Apr 1 '11 at 20:55
  • If the goal is to segment iSCSI traffic... go for it. Put it on its own unrouted network segment, be it a vlan or dedicated hardware. Enable jumbo frames, dedicate interfaces, and don't have to worry about accidentally disconnecting the wrong interface of the three that, on their face, appear to be identical. Apr 1 '11 at 21:25
  • @Bittrance: I'm fine for the traffic only using its interface. If I could just bond the interfaces and have roughly 3Gb of bandwidth available for all my local traffic I would, but bonding the interfaces doesn't increase my iscsi throughput from 1Gb. I have to keep the interfaces separate and use multipathing for that.
    – sciurus
    Apr 1 '11 at 21:59
  • @Shane: If I had control of the networking equipment, I would do that. I'm trying to do the best I can without that access.
    – sciurus
    Apr 1 '11 at 22:22

There are some things to take care about when you're multi-homing hosts. First thing is that you need to be aware of the way the Linux TCP/IP stack will handle multiple interfaces being in the same subnet regarding ARP queries and answers - this setting is the interface's arp_filter value that you can query via sysctl(1) or the /proc file system.

0 - (default) The TCP/IP stack will respond to ARP requests with addresses from other interfaces. This may seem wrong but it usually makes sense, because it increases the chance of successful communication. IP addresses are owned by the complete host on Linux, not by particular interfaces.

1 - Allows you to have multiple network interfaces on the same subnet, and have the ARPs for each interface be answered based on whether or not the kernel would route a packet from the ARP'd IP out that interface. In other words it allows control of which NICs will respond to an ARP request and will finally let your TCP/IP flows run.

You should first enable arp_filter on all the interfaces that are in the same subnet and you could easily add an entry to your routing table for your iSCSI portal to use a specific iface and adjust the other interfaces metric to so that one is prefere one over the other.

Another option is to setup source based routing as the default destination based routing will act exactly as you described wrt all ifaces being in the same subnet.

The reason why eth1 is taken out is that it's IP has the lowest numerical value and is such chosen for communication with that network.

  • Based on my understanding of kernel.org/doc/Documentation/networking/ip-sysctl.txt, I think I've handled ARP properly by setting arp_ignore to 1 and arp_announce to 2. I'm not familiar with adjusting the metric for an interface or route, can you go more into that? Also, if you can explain how source routing applies in this situation that would be helpful. I've been reading LARTC but it hasn't clicked yet.
    – sciurus
    Apr 1 '11 at 22:43

To solve my problem I wrote nethook, a daemon that runs scripts when network interfaces change state on RHEL-based distributions. I have it run this script for interfaces whose route's metric I want to increase.

EDIT: I wrote nethook before I was aware of ifup-local and ifdown-local. You can probably use them instead.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.