I have about 30 servers connecting into a Cisco 3500 series switch, with 100M upstream bandwidth. Periodically, one or more of my servers will do a large file transfer which will completely saturate the network and cause 10-15% packet loss to other devices. I need a good solution to 1) monitor my bandwidth on a per-IP Address basis across all the servers 2) Prevent any given IP from consuming more than 50% of the available bandwidth 3) Give priority to certain protocols (streaming traffic, SSH) and lower priority to other traffic (BitTorrent, SMTP, FTP). I can monitor traffic on a per-port basis with that switch, but that does not give me any IP level visibility and no bandwidth control besides setting the port speed.
I have a spare Dell R200 with dual 1G NICs that I can devote to the cause and I'm fairly comfortable around Linux. I know this can be done with IPTABLES, TC and HTB, but I would rather have a more packaged solution with decent documentation and a management interface. If the answer is buy Cisco XXX, I'm open to that also, but since I have a server sitting around, I though I'd try that first.
So the question is: What Linux/BSD Firewall/Router/QOS solution would be meet these requirements? I've seen the list at http://en.wikipedia.org/wiki/List_of_router_or_firewall_distributions, but I'm looking for real-world experience with any of them.