18

I have a linux box with 9 NICs on it, and I want eight of them to have unique addresses on the same subnet, e.g.:

ifconfig eth1 192.168.123.1 netmask 255.255.0.0
ifconfig eth2 192.168.123.2 netmask 255.255.0.0
ifconfig eth3 192.168.123.3 netmask 255.255.0.0
...
ifconfig eth8 192.168.123.8 netmask 255.255.0.0

The default ARP behaviour is extremely counterproductive in this case, since it results in all traffic for all IPs passing exclusively through eth1, which is pretty much the exact opposite of what I want.

So I rummaged around and ended up making some changes to sysctl such as this:

net.ipv4.conf.all.arp_filter=1
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

That did prevent eth1 from impersonating all the others, but I still can't ping anything other than eth1's address successfully. (e.g. from a second computer on the same switch, only 192.168.123.1 responds to ping)

I'm guessing that I need to do something with arptables or iproute or SOMETHING, but I'm lost at sea in this field.

Bonus Points: Solution must be compatible with Linux 2.6.27.27. (More specifically, Slax 6.1.2)

8
  • Can you post the routing table?
    – ponsfonze
    Aug 8, 2012 at 0:40
  • 2
    What is your purpose in creating this configuration? What are you trying to accomplish? Aug 8, 2012 at 1:03
  • this way madness lies.
    – Sirex
    Aug 8, 2012 at 3:51
  • 1
    @DavidSchwartz I am testing some networking middleware which requires a homebrew test client. Basically the middleware and test client(tx) are on Box A, and test client(rx) is running on Boxes B, C, D, ...etc. Instead of having nine boxes with one NIC each (Boxes B, C, D, etc.), I want to have one box with nine NICs because it makes a lot of things ($$$, log collection, automation, etc) easier. Additionally, there are some bandwidth and other considerations which make virtual interfaces a poor choice. E.G. in my actual test setup I will have more than one "Box A". Aug 8, 2012 at 20:12
  • 2
    @DavidSchwartz No, the machine is not supposed to act like a switch (or router). Think of it more as running 8 virtual machines with a dedicated physical NIC for each VM (Note: I am not running VMs, this is just an analogy). From the point of view of another box on the same network, my single PC should be utterly indistinguishable from eight discrete PCs. Aug 8, 2012 at 22:17

5 Answers 5

19

You need a strong end system model. Linux is fundamentally built around a weak send system model, so it's really not a good OS choice for this application.

You will have to fake every piece of the behavior you need, from ARP to policy routing to source address selection. You will also need filters to prevent packets from being accepted if they arrive on the wrong interface.

The definitely necessary steps are:

  1. Configure arp_filter=1 and arp_ignore=2 on all interfaces.

  2. Add per-interface, source-based routing for outgoing traffic. (Destination interface must be chosen based on source address.)

  3. Add per-interface ingress filtering to silently drop packets received on the wrong interface. (Packets with a destination address assigned to another interface.)

Unfortunately, there is no consensus on whether these three steps are all that is needed. The weak end system model is built into the entire Linux TCP/IP stack, and it's not clear what might go wrong with subtle issues like multicast.

It's not clear how you would choose the output interface for broadcasts, for example. Should it go out all of them? Maybe. What is the right behavior if the stack gets an outbound broadcast with a source address not assigned to one of the interfaces?

Again, you have chosen the wrong tool for the job.

3
  • What would be the right tool for the job?
    – woky
    Aug 9, 2020 at 10:55
  • @David Schwartz changing operating systems may not be practical. And it seems there is support in Linux for such a thing; see sysctl explorer at sysctl-explorer.net/net/ipv4/arp_filter: "1 - Allows you to have multiple network interfaces on the same subnet, and have the ARPs for each interface be answered based on whether or not the kernel would route a packet from the ARP’d IP out that interface (therefore you must use source based routing for this to work)." Linux may not be the best tool for the job, but it may be the tool one has. And it may work, no?
    – Mike S
    Mar 11 at 16:15
  • @MikeS As my answer explains, this fixes some of the problems but not all of them. It may work until it doesn't. Mar 11 at 23:54
7

You more likely want to create a bridge with the 8/9 interfaces and then assign an IP address to that bridge (bridge-utils packet, command 'brctl add').

This way the bridge will act like as a switch and can have an IP address into your subnet.

1
  • AFAIK, this should actually be the correct answer for Linux. Linux gets around many of the problems mentioned in David's answer by using bridge devices (aka avoid L3 problems by building up a better L2 logical network).
    – Dave
    Apr 26, 2016 at 1:15
4

I would recommend bonding the physical interfaces then configure all of the addresses on the single bonded interface.

You'll need support on the switch as well.

Here's a mini tutorial that you can use to get started.

4

It seems you want a test environment equivalent to 9 separate machines, and believed that 9 interfaces on one machine could emulate that. In Linux it simply cannot do this through a single stack for reasons David Schwartz described. BTDT and have the scars. It was bad enough with 2 interfaces.

A better solution might be to run 8 or 9 discrete virtual machines in your one host, and bridge 8 or 9 interfaces to these virtual machines.

1
  • Reading resources: Unix Network Programming: The Sockets Networking API, by Stevens, Fenner, and Rudoff. Also see RFC1122 and RFC4907.
    – Skaperen
    Aug 9, 2012 at 1:40
0

Yes it is possible following David Schwartz suggestion:

echo -ne 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo -ne 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter
echo -ne 0 > /proc/sys/net/ipv4/conf/eth3/rp_filter

//For proper functionality i.e. ARP replies from eth1 to get generated when both eth0 and eth1 are in same subnet

echo -ne 0 > /proc/sys/net/ipv4/conf/all/arp_filter
echo -ne 2 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo -ne 0 > /proc/sys/net/ipv4/conf/eth0/arp_filter
echo -ne 2 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
echo -ne 0 > /proc/sys/net/ipv4/conf/eth1/arp_filter
echo -ne 2 > /proc/sys/net/ipv4/conf/eth1/arp_ignore

//Create a table called "new_rt_table" and create a routing rule that says any packet with a mark equal to '1' gets routed according to the "new_rt_table"(can name it whatever you want) table. The file /etc/iproute2/rt_tables is the only source of table names on the system. Internally, routing tables have integer identifiers.

echo 1 new_rt_table >> /etc/iproute2/rt_tables
ip rule add from all fwmark 1 table new_rt_table

//setup the "new_rt_table" table to route the packets via eth1

ip route add default dev eth1 table new_rt_table
ip route show table new_rt_table

//mark packets so that 'ip route' can route it through eth1

iptables -F -t mangle
iptables -t mangle -I OUTPUT -s <ip addr of eth1> -o eth0 -j MARK --set-mark 1

//enable support for multiple routing tables in kernel config.

Kernel Configuration

→ Networking support → Networking options

[*] IP: advanced router

[*] IP: policy routing

CONFIG_IP_ADVANCED_ROUTER

CONFIG_IP_MULTIPLE_TABLES

//the above steps redirects packets meant to be output from eth0 to properly exit from eth1.

Kindly suggest any other methods if anyone gets it working.

1
  • @@noel_av are you sure you want 0 in arp_filter? According to sysctl explorer at sysctl-explorer.net/net/ipv4/arp_filter: "1 - Allows you to have multiple network interfaces on the same subnet, and have the ARPs for each interface be answered based on whether or not the kernel would route a packet from the ARP’d IP out that interface (therefore you must use source based routing for this to work)."
    – Mike S
    Mar 11 at 16:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .