3

When trying to set up an OpenBSD router I've run into an apparent routing problem.

I have a 1U machine with 6 gigabit NICs (em0-em5). My ISP provided me with the following:

xx.xx.97.246/28 static WAN IP.
xx.xx.97.241 default gateway address.
xx.xx.98.192/29 static block that is externally routed to via xx.xx.97.246

This is my setup:

+-------------------+
|        em0        +----> ISP's gateway (xx.xx.97.241)
|    xx.xx.97.246   |
+-------------------+
|        em2        +----> DMZ network (xx.xx.98.194-198)
|  xx.xx.98.193/29  |
+-------------------+
|        em4        +----> LAN2 (172.16.1.0/24)
|     172.16.1.1    |      [NATs to xx.xx.97.246 address]
+-------------------+
|        em5        +----> LAN (192.168.1.0/24)
|    192.168.1.1    |      [NATs to xx.xx.97.246 address]
+-------------------+

Forwarding has been enabled in sysctl.conf.

With this setup and some basic PF rules I have been able to get basic Internet access from the LAN networks (em4/5) and I can access mail/DNS servers on the DMZ network from the LAN networks. So far, so good.

The problem is in accessing the DMZ network from the Internet and any host on the DMZ network accessing the Internet (to perform upstream DNS look ups, etc). There does not seem to be any connectivity to the Internet in either direction when a DMZ net address is the endpoint.

I can ping the xx.xx.97.246 WAN address from the DMZ net but I cannot ping the ISP's default gateway address (xx.xx.97.241).

I get an odd trace when trying to traceroute from DMZ net to the WAN IP - it skips the em2 hop and only reports the WAN IP as if it was the next hop. I can trace to em2 and get it's address as the next hop if I specify the em2 address as the destination address. Regardless I cannot ping nor trace to the gateway address - I get a "host unreachable" message.

This is what I have tried so far:

  1. Adding a static route from xx.xx.98.193 to the WAN IP and got "route exists" ok, makes sense. Out of desperation I tried adding a route from xx.xx.98.193 to the gateway address and got "destination unreachable".

  2. Adding xx.xx.98.193 as an IP alias to em0 and changing the address of em2 to xx.xx.98.194. This did not seem to have any effect on anything besides breaking all routing through the gateway.

  3. Unplugged the DMZ switch from em2 and plugged in a PC configured to static address xx.xx.98.194 then disabled pf with pfctl -d which essentially passes everything and takes pf out of the picture. No change.

  4. Double-checked the arp table with arp -a and can see that em2 (it's MAC) is bound to xx.xx.98.193/29. Also checked the routing table with netstat -ranf inet and it shows the xx.xx.97.241 gateway as the default.

At this point I think this is purely a routing problem especially after taking steps 3 & 4. By the way, steps 1 & 2 were reversed and the system rebooted to revert back to the previous config before trying 3 & 4.

I've considered bridging em0 and em2 but it just doesn't seem necessary since all the interfaces are in the same machine and once an em2 host's packet makes it into the router the routing table should have the path out to the default gateway.

I also noticed an odd arp message repeating in /var/log/messages "attempt to overwrite permanent entry xx.xx.98.193 by ". The mac via arp was from the ISP's gateway xx.xx.97.241. Not sure if this is normal "chatty logging" or if it is a symptom of my problem.

Thanks in advance for your time reading and responding,

Rob

UPDATE:

Here is my routing table:

Routing tables

Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
default            xx.xx.97.241       UGS        4       54     -     8 em0  
127/8              127.0.0.1          UGRS       0        0 32768     8 lo0  
127.0.0.1          127.0.0.1          UHl        1        0 32768     1 lo0  
172.16.1/24        172.16.1.1         C          0        0     -     8 em4  
172.16.1.1         xx:xx:xx:xx:1b:60  HLl        0        0     -     1 lo0  
172.16.1.255       172.16.1.1         Hb         0        0     -     1 em4  
192.168.1/24       192.168.1.1        C          0        0     -     8 em5  
192.168.1.1        xx:xx:xx:xx:1b:61  HLl        0        0     -     1 lo0  
192.168.1.255      192.168.1.1        Hb         0        0     -     1 em5  
xx.xx.97.240/28    xx.xx.97.246       UC         1        0     -     8 em0  
xx.xx.97.241       xx:xx:xx:xx:fc:d4  UHLc       1        4     -     8 em0  
xx.xx.97.246       xx:xx:xx:xx:1b:5c  HLl        0        0     -     1 lo0  
xx.xx.97.255       xx.xx.97.246       UHb        0        0     -     1 em0  
xx.xx.98.192/29    xx.xx.98.193       UC         2        0     -     8 em2  
xx.xx.98.193       xx:xx:xx:xx:1b:5e  HLl        0        0     -     1 lo0  
xx.xx.98.195       xx:xx:xx:xx:d8:33  UHLc       0        4     -     8 em2  
xx.xx.98.197       link#3             UHLc       0        4     -     8 em2  
xx.xx.98.199       xx.xx.98.193       UHb        0        0     -     1 em2  
224/4              127.0.0.1          URS        0        0 32768     8 lo0 

I tried a lot of configs last night (primarily because I'm running out of ideas). As per the previous attempts, PF was disabled with pfctl -d to remove it as a variable. Probably the most sane attempt at something was this:

  1. Unplugged em4 and em5 (the dynamic nets) and removed their hostname.if files from /etc to leave only em0 and em2.
  2. Configured em0 with the xx.xx.98.193 address and em2 with the xx.xx.98.194 address then a PC connected to em2 with xx.xx.98.195.

With that configuration from the PC I could ping up to the 194 address (em2 IF) but not the 193 address (em0 IF). From the OpenBSD system I could ping the 194 and 195 addresses.

It seems like whatever the problem is the breakdown in routing is occurring when em2 tries to reach em0. Also for at least one ping attempt the PC made it up to em0 with 50 loss but consistently failed thereafter.

There must be something silly that I am missing here. My next crazy thing to try will be to enable/add kernel routing logs and compare what is output between successful routing between interfaces and between em0/2. Its going to be a crash course in OpenBSD's networking stack/kernel for that. :/

1 Answer 1

1

Post the output of netstat -ranf inet. What happens if you disable pf and ping a host on the Internet that’s known to respond using ping -I xx.xx.98.193? I also find your ISP providing a MAC address for xx.xx.98.193 unusual—it leads me to believe your problem is on your ISP’s end.

3
  • Please see the edited version of the question for the routing table. If I try to ping with ping -l xx.xx.98.193 I cannot even reach the ISP's upstream gateway address which is the first hop outside of our network. Something seems to be broken between em2 and em0 interfaces.
    – Kubitz
    Jan 28, 2016 at 4:27
  • With the update to your OP, your “sane” attempt at something was not really sane—you likely had the same subnet on multiple interfaces and this is likely to cause problems. If you wanted to try something “sane,” you could connect a PC to em0 and give it the IP of your ISP’s gateway (make certain your subnets are correct and the PC’s gateway would be xx.xx.97.246). Assuming everything works as expect in this configuration, I’d likely blame my ISP.
    – Bink
    Feb 2, 2016 at 2:54
  • 1
    This turned out to be an issue on the ISP's end per your original supposition. Their upstream gateway "took" the xx.xx.98.193 address which was designated as my router's IP for my static block. One quick call and the ISP fixed it and my config started working as expected. Thanks for your time in looking at this issue.
    – Kubitz
    Feb 16, 2016 at 3:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .