We've done exactly this for critical infrastructure for many years. We take three full upstream BGP feeds through Quagga's
bgpd and it uses a whopping 658MB of RAM to run the whole system. For this purpose Debian is much more solid than other OSs in our experience (and it also needs less security updates with its minimum install footprint, causing much fewer reboots than the two other OSs we've tried). We use Ksplice so we only boot for critical package updates. Don't worry at all about compatibility with other vendors at your ISP ... RIPE the RIR use Quagga !
Surprisingly the hardware isn't that important, it's all about the NICs. Fast CPUs basically just mean the prefixes load quicker if you refresh the sessions (assuming you've got a GB of RAM and they load into memory) so an entry-level Quad Core is massively over-specced. We spent a long time trying different NICs and in our experience the best are the Intel cards which use the
igb driver (for about £100/NIC we use the: 82576, ET Dual Port Server Adapter) with the
e1000 coming second. There are a few considerations like how your ingress and egress NICs talk to the mainboard but for sub 250Mbps you probably won't notice if you use these NICs. We've repelled a sophisticated UDP DDoS attack using this architecture (it used the tiniest UDP packets which routers struggle to handle). Bear in mind being able to process the highest number of packets is what you're most concerned with and not necessarily the throughput, measured in Mbps. For very little money we've specified a Gigabit multihomed router that can handle standard Internet size packets, ie normal operation, up to 850Mbps !
I started with Cisco (
bgpd's config is near-enough identical so if you've got experience with Cisco kit it's a really quick transition) but because Linux is so malleable (eg being able to add a few low-resource scripts to your routers to help with reporting and admin) IMHO makes it incredibly powerful (and underrated) for this type of set up. You can't go far wrong reading some of the Nanog Mailing list archives if you're still in any doubt or need further help.
This should get you started pretty quickly on Debian: Easy Quagga Tutorial