I have a server with multiple interfaces and IP addresses on the same network. I would like for non-iSCSI traffic to exclusively use one of the interfaces, and for iSCSI traffic to exclusively use the rest of the interfaces. Limiting the iSCSI traffic to a subset of my interfaces is easy; I simply don't create entries in /var/lib/iscsi/ifaces/ for the interfaces I don't want it to use. However, I'm not sure what is a good approach to limiting the non-ISCSI traffic to one interface. As far as linux is concerned, the iSCSI and non-iSCSI interfaces are equally good routes to the network.
Here's an example configuration
The iSCSI storage has the IP addresses 172.16.50.70-78.
The server has the following interfaces, addresses, and routes.
$ ip route list 18.104.22.168/24 dev eth0 proto kernel scope link src 22.214.171.124 172.16.0.0/16 dev eth1 proto kernel scope link src 172.16.50.80 172.16.0.0/16 dev eth2 proto kernel scope link src 172.16.50.81 172.16.0.0/16 dev eth3 proto kernel scope link src 172.16.50.82 default via 126.96.36.199 dev eth0
The desired configuration is for eth3 to be used for non-ISCSI traffic and for eth1 and eth2 to be used for iSCSI traffic. However, non-iSCSI traffic currently goes out eth1.
$ ip route get to 172.16.50.90 172.16.50.90 dev eth1 src 172.16.50.80
(some edits since original posting below)
With my current configuration, if eth1 and eth2 are fully saturated sending iSCSI traffic, my non-iSCSI traffic will compete with the iSCSI traffic on eth1 while eth3 sits idle.
How could I configure linux to prefer to send traffic to the local network using eth3 rather than eth1 or eth2?
I have already set net.ipv4.conf.all.arp_ignore to 1 and net.ipv4.conf.all.arp_announce to 2. This should prevent my IP addresses from floating between the interfaces, e.g. arp flux. I think I just need help with the routing.
Thanks to pfo I started looking at metrics. If I delete the routes and recreate them with the iSCSI interfaces having a higher metric than the non-iSCSI interface, things seem to work the way I want. iSCSI traffic still uses the dedicated interfaces without me setting up static routes to the iSCSI IP addresses. All other local traffic goes out eth3. Now I need to figure out the proper way to set the metrics automatically when the interfaces are brought up. This is on RHEL 5.5.
ip route delete to 172.16.0.0/16 dev eth1 ip route delete to 172.16.0.0/16 dev eth2 ip route delete to 172.16.0.0/16 dev eth3 ip route add to 172.16.0.0/16 dev eth1 src 172.16.50.80 metric 1 ip route add to 172.16.0.0/16 dev eth2 src 172.16.50.81 metric 1 ip route add to 172.16.0.0/16 dev eth3 src 172.16.50.82 metric 0
Assigning a different metric using the existing RHEL network scripts seems impossible, https://bugzilla.redhat.com/show_bug.cgi?id=498472