14

Hello networking gurus,

I've got a Linux (kernel 3.14) server which acts as a TFTP, NFS and HTTP server for a farm of consumer electronics devices (set-top boxes - STBs). The devices use TFTP to boot their kernels from, then mount their root FSes from the NFS server on our machine, etc etc.

Now, for one esoteric technical reason I am not going to delve in here (just believe me:) , each STB has to be in its own, physically separate, LAN. So the way networking is set up ATM is:

The server has 1 network card which is used to access the rest of the world. It also has 1 network card for each STB it serves - and each of those is connected to a small router, to which the STB + some other devices are connected and form a LAN.

Currently there are 3 STBs connected, and the LANs are 172.16.50.0/24, 172.16.51.0/24 and 172.16.52.0/24. Its all working nicely.

However: the fact that we have 3 different LANs means the very same server has to be accessed as 172.16.50.1 from STB1, 172.16.51.1 from STB2 and 172.16.52.1 from STB3 - and that means that we have a little bit different environment on each STB and each time we - say - upload new RootFS to be used on the STBs, we need to manually edit some configuration file and put the correct IP the server has to be accessed from this particular STB. Not very convenient and error-prone!

That got me thinking: what if we simply configured those three LANs all to be the very same 172.16.50.0/24? From the STB (and rest of the devices in the LAN) point of view everything should be fine, but what about the server's point of view?

Can a Linux server have N different ethernet interfaces, all configured with the same static IP, but each connected to a physically separate LAN?

8
  • 3
    Maybe sharing the esoteric reason (even though i believe you there is a reason) could give us a hint in a direction of a solution, because having three separate network cards with the same IP is probably not going to do any good, even if you use a few tricks to get it working. For example: do those STBs have the server as default gateway, as that could be a solution.
    – Fox
    Jul 15, 2015 at 13:09
  • No, all the STBs do with the server is boot their kernels and mount their RootFSes from it. Having booted, they access the internet through a totally different way and do not need the server at all.
    – Leszek
    Jul 15, 2015 at 13:11
  • Sharing the esoteric reason would not be easy I am afraid, because there's currently an email thread with 3207 emails from 113 different people from 4 different companies about this esoteric 'bug-slash-feature'. The thread has been going on for 1.5 years. Yes, I downloaded the .odf file form Outlook and wrote a Python script to parse it and come up with those numbers, because I had a feeling it might be a record-breaker.
    – Leszek
    Jul 15, 2015 at 13:13
  • Actually I was not interested in knowing whos fault it is, or the specific reason inside the STB. I was interseted in knowing something like - broadcasts from one STB kill the other, or something like that. Can you influence routing tables on the STBs?
    – Fox
    Jul 16, 2015 at 9:01
  • 1
    Well this leads me to extending the solution by Cha0s, by adding ebtables inbetween to filter traffic from one STB to another. Bridge alone would lead to the situation, where the STBs see each other. This can be combined with managed switch and VLANs. You just need one VLAN per STB. And it may be easier to use than namespaces.
    – Fox
    Jul 17, 2015 at 17:51

3 Answers 3

22

Yes this is possible, using a nice feature called network namespaces (see man ip-netns(8)). It basically gives you multiple distinct network stacks, each with its own set of interfaces, routes etc.

You would need to create a namespace for each of your STBs and could then run your required services separately in each namespace.

For the namespaces you would need to proceed as follows:

  • Create a namespace called net1:

    ip netns add net1
    
  • Assign your interface ethX to the new namespace and configure your IP address 172.16.50.1:

    ip link set dev ethX netns net1
    ip netns exec net1 ip link set dev ethX up
    ip netns exec net1 ip address add 172.16.50.1/24 dev ethX
    

The IP address 172.16.50.1 is now not visible from your default namespace. A simple ping 172.16.50.1 doesn't work, you first need to switch to the net1 namespace and execute the command there:

ip netns exec net1 <command>

In this way you can now run each service in each of your namespaces.

If you feel adventurous, you could even try to somehow redirect all requests from your STBs to a central service. For this you need a tunnel from each namespace to your default namespace (see ip link help veth) and quite some iptables magic...

5
  • I love how you can learn new things on SF :D Thanks for your info! Really useful! Is this feature related to VRF? (kinda sounds like it)
    – Cha0s
    Jul 16, 2015 at 15:56
  • @Cha0s I don't know about VRF on Linux, but for sure netns looks the same as VRF on routers (though it is probably not the same performance-wise).
    – Oliver
    Jul 17, 2015 at 4:58
  • 1
    Excellent! Now I hope you can combine that idea with another: a managed switch. I'd like to get rid of the multiple network cards. Setup 3 VLANS id 2,3,4, one VLAN for each STB, setup a trunked port to the server, three interfaces eth1.2, eth1.3, eth1.4, each one in its own network namespace and with the same static IP ? That would be sweet.
    – Leszek
    Jul 17, 2015 at 10:40
  • Oh, so I either start a separate instance of TFTP, NFS and HTTP server on each namespace, or I delve deep into tunneling stuff from the per-STB namespaces to the default one? Hmm...
    – Leszek
    Jul 17, 2015 at 11:03
  • I'm sure it also works with the VLAN so you don't need a separate network adapter for each of your STBs. As for your other question, the tunnel shouldn't be too hard to implement. I would start with a routed solution with just one STB network and then experiment with iptables on either end of the tunnel. I'm no iptables expert so I don't have the required commands ready but I'm sure you'll manage (or maybe come up with a followup question on SF).
    – Oliver
    Jul 17, 2015 at 12:32
3

Technically you can, but configuration would be colorful. It would be a kludge and it would require each STB to have a unique IP while being isolated on their own wire to satisfy your requirement. Configuration on the clients doesn't change. Here's the config on the server:

ifconfig eth0 10.0.50.1 netmask 255.0.0.0
route del -net 10.0.0.0 netmask 255.0.0.0 dev eth0
route add 10.0.50.2 dev eth0
ifconfig eth1 10.0.50.1 netmask 255.0.0.0
route del -net 10.0.0.0 netmask 255.0.0.0 dev eth1
route add 10.0.50.3 dev eth1
# ...

This should result in only one IP per interface in the routing table. These are on separate wires, so there's no crosstalk. Each of these devices would think they were on a 2 node 10.0.0.0/8.

When the server wants to talk to 10.0.50.2, it will look at the ARP table and then routing table. If the ARP table is empty, the routing table tells it to send an ARP request on the respective interface, therefore they MUST have unique IPs or the server will only be able to speak to the last route added.

You can set your DHCP server to assign IPs based on the hardware addresses or run a separate dynamic DHCP range on each interface. The DHCP server can hand out anything, but 10.0.50.3 on the eth0 wire will not be reachable.

-1

You cannot use the same IP address on multiple interfaces. It just won't work properly (usually it will only work on the last interface the IP was assigned on).

You need to put the ethernet interfaces into a bridge and assign the IP address on the bridge itself.

Essentially all ethernet ports in that bridge will work as a Switch.

Alternatively you can remove all ethernet cards for each STB and simply add a switch (which is more scalable than adding new ethernet cards on your server).

But since there is a requirement for each STB to be on its own broadcast domain then I am afraid you need to stick to your current setup.

Or to simplify at least the server hardware part of your setup, ditch the multiple ethernet cards and simply add a managed switch and use VLANs to simulate the 'multiple ethernet cards' using only one physical ethernet card.

2
  • We were thinking about a managed switch and VLANs, and most probably that's what we are going to do once we have to add new STBs. In the meantime, I'll try your 'bridge' idea. Thanks!
    – Leszek
    Jul 15, 2015 at 13:37
  • Downvoting because technically all IPs are on all interfaces under Linux. The attachment of an IP to an interface is useful when, for example, the interface is removed it can automatically remove the IP. The "from" address on outgoing connections is automatically selected based on the route, not the interface. So beyond that all interfaces will use the all IPs. Oct 23, 2023 at 20:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .