Properly configure two eth interfaces on different subnets

Question

Can anyone with heavy background in both Linux and Networking help me out? I've scoured the Internet looking for a clear answer for this and I've even compared a box I have access to with the same setup and I can't get my box working.

The basic situation is this:

===========================

A RHEL 6 Linux box has two Ethernet interfaces:

eth0 = 10.10.5.10/24
eth1 = 10.10.6.10/24

[root@box ~]# netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.10.5.0      0.0.0.0         255.255.255.0   U         0 0          0 eth0
10.10.6.0      0.0.0.0         255.255.255.0   U         0 0          0 eth1
0.0.0.0        10.10.5.254     0.0.0.0         UG        0 0          0 eth0

ip forwarding has been enabled and is active.

The problem:

Clients on the same subnet as eth1 or the 10.10.6.x network can ping and telnet to resources running on eth1 on this box.

However, clients on any other subnet cannot. So from my workstation I can ping and telnet to resources on the box's eth0 all day long, but nothing on eth1... What gives?

Comparing this setup to another box of a similar nature in my environment shows no discernible differences.

Here are some files:

# cat ifcfg-eth0
DEVICE="eth0"
BOOTPROTO="none"
HWADDR="Omitted"
IPADDR="10.10.5.10"
NETMASK="255.255.255.0"
NETWORK="10.10.5.0"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="no"

# cat ifcfg-eth1
DEVICE="eth1"
BOOTPROTO="none"
HWADDR="omitted"
IPADDR="10.10.6.10"
NETMASK="255.255.255.0"
NETWORK="10.10.6.0"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="no"

# cat route-eth0 
default via 10.10.5.254 dev eth0

# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=box
GATEWAY=10.10.5.254
NOZEROCONF=yes

Any assistance you can provide will be appreciated!

Thanks.

Adding more info, but with brevity included:

# ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    inet 10.10.5.10/24 brd 10.10.5.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    inet 10.10.6.10/24 brd 10.10.6.255 scope global eth1

Answer 1

with the setup as you currently show:

• you have 2 sides :

  • the eth1 side (10.10.6.10 interface, on 10.10.6.0/24 lan)
  • the eth0 side (10.10.5.10 interface, on 10.10.5.0/24 lan)
  • you defined that machine's default gateway on eth0 side (default gw ip: 10.10.5.254)

• therefore

  • on eth1 side:
    • [requisite] any machines in 10.10.6.0/24 should be able to communicate with 10.10.6.10 (and with each other as well) (I assume no firewalling issues)
    • [requisite] in addition: any machine in 10.10.6.0/24 THAT HAS defined their default gateway as "10.10.6.10" will be able to send packets to other machines, going through your machine (10.10.6.10/24->eth1[machine]eth0-> other lans). Actually you don't need to have it as default gateway, but for ANY lan on the eth0 side, they need to associate that LAN with the 10.10.6.10 gateway.
  • on eth0 side:
    • [requisite] any machine on 10.10.5.0/24 should see 10.10.5.10 (eth0) and can communicate with it (and with each other).
    • [requisite] any machine on another LAN should be able to communicate with eth0, via (with maybe additionnal hops) the machine which has 10.10.5.254 ip (ex: if that machine have 2 subnets, 10.10.5.254/24 on the same side as eth0, and 10.x.y.z/24 on another interface, machine in 10.x.y.0/24 can send packets to 10.10.5.10 by having a route: 10.10.5.0/24 with gateway 10.x.y.z).
    • if so, then they just need to have an additionnal route to reach 10.10.6.0/24 : 10.10.6.0/24 via gateway 10.x.y.z

it can get complicated to put in words...

what would help: tell us the list of networks ( ) and each network's getways.

right now I guessed you try to have:

 [everything other machines (even internet, via the exterior internet gateway)]
                  |
                  |(links via switch(s) for 10.x.y.0/24 network, for example.
                  |   If you have such an additionnal LAN.
                  |   Maybe it's even "the Internet", and therefore not in 10.x anything,  
                  |   but this is just an example.)
                  |
                  |      
      (10.x.y.z?) |  [---------------]
   (or Internet?)ethZ[some router/box]ethW(10.10.5.254)
                     [---------------] |
                                       |
                                       |(links via switch(s) for 10.10.5.0/24 network)
                                       |
                             [-------] |
             (10.10.6.10)eth1[yourbox]eth0(10.10.5.10)
                          |  [-------]   
                          |
                          |(links via switch(s) for 10.10.5.0/24 network)
                          |
                          |  [---------------------------]
                         ethx[any machine in 10.10.6.0/24]
                             [---------------------------]

General way to debug your setup:

• imagine you ARE the IP packet: you have a source IP, a destination IP.
• Now, on the interface you sit on at the source, see what that machine's routes tells you which next hop is. And go there(but first ensure that machine's setup allows you to, for example on the eth0/eth1 machine, it needs ipforward to go from one interface to the other). And proceed, one hop at a time, toward your goal (AND back to your source : sometimes the destination machine's route/defaultgw are broken and therefore the reply packet is not sent via the right path)
  • ex1 (very simple): 10.10.6.10 trying to send to 10.10.5.10: step1 you are on eth1 (10.10.6.10). That machine has a route saying "10.10.5.0/24 is directly here, on eth0". Ipforwarding allows you to hop from eth1 to eth0. Proceed to eth0. step2: you are now (same IP packet) on eth0. It turns out this is your destination: GOOD. step 2b: reply packet: inverse source/destination : you are now a packet (10.10.5.10 -> 10.10.6.10=): thank god, a local route tells you the next hop is on eth1. step4: no eth1 : you are at destination. GOOD.
  • ex2 (a bit longer ex:) you are a packet from "any_machine", 10.10.6.12 (on the eth1 side). you try to reach as destinatino google (8.8.8.8, for example). So you are "10.10.6.12 -> 8.8.8.8" and at first you sit on the 10.10.6.12 interface (say, eth3). A local route on "any_machine" tells you (as it was the prerequisite, above): "default gw=10.10.6.10", so you hop to "yourbox", and arrive there on the eth1 side. step2: you are now on "yourbox/eth1" : now local routes tell you "you need to go to default gw: 10.10.5.254, via eth0" (and ipforward on "yourbox" allows you to hop from eth1 to eth0). Now you arrive on eth0, and following the default gw route, you arrive later on "somerouter" on ethW(10.10.5.254). etc... (and don't forget to check the way-back route)

Answer 2

Your default gateway is at 10.10.5.254, however the eth1 interface is configured as 10.10.6.10/24. What this effectively means is that the 10.10.6.x interface cannot hit the default gateway as they are on different subnets.

More general network information is needed to properly assess your options, however your two options from what has been posted so far are to:

• Confirm that the two networks are in fact /24 subnets, if so, set the proper gateway for eth1 in that interface's config file
• If the interfaces are part of a larger supernet, configure the two interfaces to exist on a /16 subnet to use the same gateway.

Answer 3

Have you enabled IP forwarding?

sysctl net.ipv4.ip_forward

If it says 0 you have not and your box is not routing packets.

Enable it by

sysctl -w net.ipv4.ip_forward=1

And make it permanent by adding the following to /etc/sysctl.conf:

net.ipv4.ip_forward = 1