26

We have an OpenBSD router at each of our locations, currently running on generic "homebrew" PC hardware in a 4U server case. Due to reliability concerns and space considerations we're looking at upgrading them to some proper server-grade hardware with support etc.

These boxes serve as the routers, gateways, and firewalls at each site. At this point we're quite familiar with OpenBSD and Pf, so hesitant at moving away from the system to something else such as dedicated Cisco hardware.

I'm currently thinking of moving the systems to some HP DL-series 1U machines (model yet to be determined). I'm curious to hear if other people use a setup like this in their business, or have migrated to or away from one.

1
  • 1
    I found the answers helped us as we have been running open bsd for 9 years and began to think to move to jos because of power issues in the data centre. Now I will think again as I think we have undervalued the benefits of running on an open platform.
    – user105680
    Jan 2, 2012 at 11:09

11 Answers 11

44

We run exclusively OpenBSD routers/firewalls to serve FogBugz On Demand. Unless you're operating in a transit role and need the extremely high pps throughput that purpose-built hardware and integrated software can provide, OpenBSD on solid hardware will be a more manageable, scalable, and economical solution.

Comparing OpenBSD to IOS or JUNOS (in my experience):

Advantages

  • The pf firewall is unmatched in terms of flexibility, manageable configuration, and integration into other services (works seamlessly with spamd, ftp-proxy, etc). The configuration examples do not do it justice.
  • You get all the tools of a *nix on your gateway: syslog, grep, netcat, tcpdump, systat, top, cron, etc.
  • You can add tools as necessary: iperf and iftop I've found very useful
  • tcpdump. Enough said.
  • Intuitive configuration for Unix veterans
  • Seamless integration with existing configuration management (cfengine, puppet, scripts, whatever).
  • Next gen features are free and require no add-on modules.
  • Adding performance is cheap
  • No support contracts

Disadvantages

  • IOS/JUNOS make it simpler to dump/load an entire configuration. Absent any configuration management tools, they will be easier to deploy once your config is written.
  • Some interfaces simply aren't available for or stable on OpenBSD (e.g., I know of no well-supported ATM DS3 cards).
  • High-end dedicated Cisco/Juniper-type devices will handle higher pps than server hardware
  • No support contracts

So long as you're not talking about backbone routers in an ISP-like environment or edge routers interfacing with specialized network connections, OpenBSD should be just fine.

Hardware

The most important thing to your router performance is your NICs. A fast CPU will quickly get overwhelmed under moderate load if you have shitty NICs that interrupt for every single packet they receive. Look for gigabit NICs that support interrupt mitigation/coalescing at least. I've had good luck with Broadcom (bge, bnx) and Intel (em) drivers.

CPU speed is more important than in dedicated hardware, but not something to fret about. Any modern server-class CPU will handle a ton of traffic before showing any strain.

Grab yourself a decent CPU (multiple cores don't help much just yet, so look at raw GHz) good ECC RAM, a reliable hard drive, and a solid chassis. Then double everything and run two nodes as an active/passive CARP cluster. Since 4.5's pfsync upgrade you can run active/active, but I haven't tested this.

My routers are running side-by-side with our load-balancers in 1U twin-node configurations. Each node has:

  • Supermicro SYS-1025TC-TB chassis (built-in Intel Gigabit NICs)
  • Xeon Harpertown Quad Core 2GHz CPU (my load balancers use the multiple cores)
  • 4GB Kingston ECC Registered RAM
  • Dual-port Intel Gigabit add-in NIC

They've been rock-solid since deployment. Everything about this is overkill for our traffic load, but I've tested throughput upwards of 800Mbps (NIC-limited, the CPU was mostly idle). We make heavy use of VLANs, so these routers have to handle a lot of internal traffic too.

Power efficiency is fantastic since each 1U chassis has a single 700W PSU powering two nodes. We've distributed the routers and balancers through multiple chassis so we can lose an entire chassis and have pretty much seamless failover (thank you pfsync and CARP).

Operating Systems

Some others have mentioned using Linux or FreeBSD instead of OpenBSD. Most of my servers are FreeBSD, but I prefer OpenBSD routers for a few reasons:

  • A tighter focus on security and stability than Linux and FreeBSD
  • The best documentation of any Open Source OS
  • Their innovation is centered around this type of implementation (see pfsync, ftp-proxy, carp, vlan management, ipsec, sasync, ifstated, pflogd, etc - all of which are included in base)
  • FreeBSD is multiple releases behind on their port of pf
  • pf is more elegant and manageable than iptables, ipchains, ipfw, or ipf
  • Leaner setup/install process

That said, if you're intimately familiar with Linux or FreeBSD and don't have the time to invest, it's probably a better idea to go with one of them.

3
  • Thanks for the extremely detailed reply. What you describe is pretty much exactly the type of system we're looking at building, a pair of servers with onboard dual GigE and and a dual GigE add-in NIC in a CARP failover configuration. It's very reassuring to see that someone else is running such a setup in a major production system. Jun 29, 2009 at 22:28
  • 1
    Personally I prefer iptables, I think pf is too restricted. My experience with CARP on OpenBSD is that it's great when you want to do planned jobs (planned failover), but the failover will most often not work when there is an actual fault. I've had exactly one successful pf crash failover, and this was with OpenBSD 4.5. Also, the support situation for OpenBSD is dismal. If you don't have the knowledge in-house or pay someone then the answer to all questions or support when it crashes is: "you mother is fat".
    – Thomas
    Jun 30, 2009 at 6:32
  • 1
    I run pf/pfsync/CARP two firewalls in a failover config. I've experienced two failover situations and in both cases I only learned about it from my monitoring system telling me one of the firewalls was down. The cluster's services continued without noticeable interruption.
    – Insyte
    Aug 12, 2009 at 15:08
8

pfsense Is a great FreeBSD based firewall, its very feature rich, easy to setup, and has an active community as well as support options. There are several people using it in commercial / production situations that are active in the forum. I use it at home and I'm pushing it at work, its a really well put together alternative. They even have a VM image for download to test it out with!

2
  • i looked at that link. that variant of MonoWall looks great. :-)
    – djangofan
    Aug 12, 2009 at 15:22
  • I believe mono focuses on embedded hardware, whereas pfsense focuses on pc based systems. I believe it was intended to offer more advanced/enterprise-class features than those found in m0n0wall or other basic firewall distro's.
    – Chance
    Sep 24, 2010 at 15:02
2

Where I work we are using RHEL5 + quagga & zebra over 4 boxes to run transit for 450mbps. So yes, you can do it in the enterprise and save a lot of money.

We do rate limiting using TC and make use of iptables and notrack rules.

2

I have used OpenBSD 3.9 as a firewall and switched to a Juniper SSG5.

As said by sh-beta OpenBSD as a LOT of good features: pf is amazing, tcpdump, lot of good tools...

I had some reasons to switch to Juniper. In particular, the configuration is fast and easy. On OpenBSD everything is "a little bit complicated".

for ex: the bandwith management is -in my opinion- a lot easier to configure on the SSG.

The OpenBSD version I used was quite old; Maybe newer version are better on this point.

1
  • On the hardware side, my OpenBSD box was an old Dell GX280.
    – Matthieu
    Aug 12, 2009 at 15:22
1

For my father's small business with one branch office, I use OpenBSD as the router/gateway/firewall for both the main and branch office. It has never let us down. We use a Dell Tower Server at each location. Each server is equipped with a Dual GiGE card, 8GB of ram (slight overkills, I know) and works well. The branch office is configured to connect to the main one through IPSEC and OpenBSD's IPSEC implementation is delightfully easy to use.

1

OpenBSD gateways are used in many enterprise setups. We have two OpenBSD gateways on our networks.

I still recall one funny episode with OpenBSD: the hard disk died, but the gateway just carried on routing traffic, like nothing had happened, serving from memory alone. It gave me some time to setup another instance.

Very low hardware requirements, Dual Opteron 248's are great. I rarely see the cpu go over 5%. They are very stable. I've being using it just over 7 years now with no issues.

1

I have been running OpenBSD (4.9) in production on our main firewall for quite some time. Its a rather old ASUS MB with 2 GB DDR (1) RAM and a dual core (2 GHz) Athlon. I bought a quad port intel card (pci-express) and used in the x16 graphics port. Do NOT throw away your PCI graphics cards if you have any laying around. You will need it as a graphics card if you plan on using the 16x PCI-express port for the NIC (onboard gfx didnt work in my case).

I know its not "Enterprise class" hardware. but these are the clear benefits of this setup:

  • I have a lot of these MB lying around, and thus will never run out of spare parts (getting ready for CARP also).

  • Most cheap AMD bords support ECC RAM!.

  • All hardware/spare parts are "of the shelf" cheap and stable

  • Performance on these rigs are great (4x Gbps), even for our rather heavy hosting setup!

0

I have in the past. I installed it originally on some "whitebox" PC's, then upgraded to a Dell Power Edge 2950. Redundant power supplies, hard drives - big improvement from a reliability standpoint. Not an observed improvement of course, we got lucky and the whitebox never crashed, but theoretically we were in better shape with more redundancy.

We were only using it to packet filter a T1, so not a noticeable performance improvement.

0

Did you considered switching to FreeBSD? OpenBSD can't fully utilize modern SMP systems (i.e Core2Quad). FreeBSD has pf and ipfw that you can use simultaneously and also has non-GIANT networking layer.

We've been running software FreeBSD routers as ISP gateways for years, this saved us a lot of $$

0

I can't speak for *BSD (yet...give me time...) but we've been running Linux routers for 10+ years and love them. Cheaper, no license hassles, and if you look at the docs you'll find you have most of the tools you need to get things done. I would suspect that BSD is very much in the same boat.

We're running a DL365 G1 with a single processor socket filled and 6Gb, although the RAM is mostly for servicing mailboxes...

0

Use Intel (em) Gigabit Server NICs.

One card that works well is the HP NC360T. It's dual port and pci-express.

Not the answer you're looking for? Browse other questions tagged .