Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I recently ran into a situation where I needed two IP addresses on the same subnet assigned to one Linux host so that we could run two SSL/TLS sites. My first approach was to use IP aliasing, e.g. using eth0:0, eth0:1, etc, but our network admins have some fairly strict settings in place for security that squashed this idea:

  1. They use DHCP snooping and normally don't allow static IP addresses. Static addressing is accomplished by using static DHCP entries, so the same MAC address always gets the same IP assignment. This feature can be disabled per switchport if you ask and you have a reason for it (thankfully I have a good relationship with the network guys and this isn't hard to do).
  2. With the DHCP snooping disabled on the switchport, they had to put in a rule on the switch that said MAC address X is allowed to have IP address Y. Unfortunately this had the side effect of also saying that MAC address X is ONLY allowed to have IP address Y. IP aliasing required that MAC address X was assigned two IP addresses, so this didn't work.

There may have been a way around these issues on the switch configuration, but in an attempt to preserve good relations with the network admins I tried to find another way. Having two network interfaces seemed like the next logical step. Thankfully this Linux system is a virtual machine, so I was able to easily add a second network interface (without rebooting, I might add - pretty cool). A few keystrokes later I had two network interfaces up and running and both pulled IP addresses from DHCP.

But then the problem came in: the network admins could see (on the switch) the ARP entry for both interfaces, but only the first network interface that I brought up would respond to pings or any sort of TCP or UDP traffic.

After lots of digging and poking, here's what I came up with. It seems to work, but it also seems to be a lot of work for something that seems like it should be simple. Any alternate ideas out there?


Step 1: Enable ARP filtering on all interfaces:

# sysctl -w net.ipv4.conf.all.arp_filter=1
# echo "net.ipv4.conf.all.arp_filter = 1" >> /etc/sysctl.conf

From the file networking/ip-sysctl.txt in the Linux kernel docs:

arp_filter - BOOLEAN
    1 - Allows you to have multiple network interfaces on the same
    subnet, and have the ARPs for each interface be answered
    based on whether or not the kernel would route a packet from
    the ARP'd IP out that interface (therefore you must use source
    based routing for this to work). In other words it allows control
    of which cards (usually 1) will respond to an arp request.

    0 - (default) The kernel can respond to arp requests with addresses
    from other interfaces. This may seem wrong but it usually makes
    sense, because it increases the chance of successful communication.
    IP addresses are owned by the complete host on Linux, not by
    particular interfaces. Only for more complex setups like load-
    balancing, does this behaviour cause problems.

    arp_filter for the interface will be enabled if at least one of
    conf/{all,interface}/arp_filter is set to TRUE,
    it will be disabled otherwise

Step 2: Implement source-based routing

I basically just followed directions from http://lartc.org/howto/lartc.rpdb.multiple-links.html, although that page was written with a different goal in mind (dealing with two ISPs).

Assume that the subnet is 10.0.0.0/24, the gateway is 10.0.0.1, the IP address for eth0 is 10.0.0.100, and the IP address for eth1 is 10.0.0.101.

Define two new routing tables named eth0 and eth1 in /etc/iproute2/rt_tables:

... top of file omitted ...
1    eth0
2    eth1

Define the routes for these two tables:

# ip route add default via 10.0.0.1 table eth0
# ip route add default via 10.0.0.1 table eth1
# ip route add 10.0.0.0/24 dev eth0 src 10.0.0.100 table eth0
# ip route add 10.0.0.0/24 dev eth1 src 10.0.0.101 table eth1

Define the rules for when to use the new routing tables:

# ip rule add from 10.0.0.100 table eth0
# ip rule add from 10.0.0.101 table eth1

The main routing table was already taken care of by DHCP (and it's not even clear that its strictly necessary in this case), but it basically equates to this:

# ip route add default via 10.0.0.1 dev eth0
# ip route add 130.127.48.0/23 dev eth0 src 10.0.0.100
# ip route add 130.127.48.0/23 dev eth1 src 10.0.0.101

And voila! Everything seems to work just fine. Sending pings to both IP addresses works fine. Sending pings from this system to other systems and forcing the ping to use a specific interface works fine (ping -I eth0 10.0.0.1, ping -I eth1 10.0.0.1). And most importantly, all TCP and UDP traffic to/from either IP address works as expected.


So again, my question is: is there a better way to do this? This seems like a lot of work for a seemingly simple problem.


Update: The solution above ended up being incomplete. Things worked fine if traffic stayed on the same subnet, but communications to other subnets using the 2nd interface would not work properly. Rather than digging a bigger hole I ended up talking with the network admins and got them to allow multiple IP addresses for the one interface and used IP aliasing (e.g. eth0 and eth0:0).

share|improve this question
    
The key distinction to remember is that the IP does not belong to the interface, it belongs to the machine. So it is correct to send out either interface for either IP in this setup, hence why it requires some trickery to not do that. –  MikeyB Nov 29 '11 at 23:24
    
I believe source routing isn't necessary in this case because the arp filtering should apply only when the switch is sending to an interface /has to find it among its ports. I could be wrong, but when the machine sends data towards the switch, the IP in the source field (IP header) isn't checked, just the arp sending the packet. –  AndreasM Dec 2 '11 at 14:09
    
MikeyB is correct, source routing is the only way. The machine can choose any outbound interface it wants to send traffic, as long as its on the same subnet. Doesnt matter if the IP isnt really located on that interface or not. –  Patrick Dec 2 '11 at 14:30
1  
AndreasM: The incoming packets aren't the problem, it's the outgoing packets which are the problem. Without source routing, all outgoing packets will use the default route, which is bound to one interface or the other. Even though the outgoing packets will have the correct source IP address, the filters on the switch will not allow them to go out since that IP address does not belong to the MAC address of that interface. To the switch it just looks like a client is trying to spoof the IP of another client. (These are smart layer 3 switches). –  Scott Duckworth Dec 2 '11 at 14:30
    
IP aliases are obsolete and a hack, do not reflect reality, not to mention that taking down the primary will take down all aliases. Use ip from iproute2 to add more than one address to the same interface. –  pilona Aug 22 '13 at 20:16
add comment

1 Answer 1

up vote 5 down vote accepted

Yes, the better way is to build a proper business case, and have them relax the rules on the switches so that you can have multiple IPs on one NIC.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.