I'm looking for recommendations on firewalls for a dedicated server that will be hosting a few websites of average traffic (5000 uniques monthly). the hosting company has port and full hardware firewalls. There's also the option of installing software firewalls. What is the level of protection of 'port' protection mentioned here: http://www.xlhost.com/firewall.php? Is this adequate? Any recommendations (Windows Server OS)?
It looks like a port on a FireBox would offer decent protection. Do you want application level protection? The FireBox probably won't be able to discern different application traffic on the same TCP/IP port. Meaning, if you had a software firewall, you could configure it just a bit more granular to only allow a certain executable to have access to port 80 (for example) whereas with a port on a firewall, you could only tweak port 80 to either allow or deny traffic without respect to what executable is initiating or responding to the traffic on your server.
As ErikA pointed out, Windows Server's built-in firewall would seem to be sufficient (as long as it was Server 2008 or above). Some alternatives that are noteworthy are:
With me not knowing exactly what your desires are, you'll have to do some digging to compare the products with the internal Windows firewall and see exactly what benefits are of significance to you.
Unfortunately, Getting Firewalls right is like getting encryption right - A lot of people think they can do it right, for that matter a lot of experts think they can get it right. Getting it right is not so easy.
If the FD mailing list is any guide, The firebox is vulnerable to Syn-Fin ambiguity. Best evidenced by this snippet of code.
if(tcp->syn && !(tcp->ack || tcp->rst)) tcpsyn=1;
If it isn't immediately obvious, A piece of processing code like this allows you to pass packets through the firewall as long as you set the Finish with the Synchronize bits thus effectively defeating port-filtering.
In short, If security is a concern for you, I'd stick with the tried and true - ASA/PIX. Failing that, BlackIce firewall is very well done, as well as pf/iptables of course.
You could also look at pfsense: http://www.pfsense.org/index.php?option=com_content&task=view&id=40&Itemid=43
Assuming its actually a dedicated server and not a VPS, you could install a bare-bones linux distro on the metal (debian-netinst, etc), run Shorewall and then run the windows machine under Xen, KVM or a VirtualBox headless instance.