Is it possible to use Windows Firewall to block specific web sites for all browsers?
Yes, you should be able to achieve that using the Advanced Settings of the firewall. But you will need to know all the ip addresses of the remote servers.
See this example which talks about ICMP but you can do the same for any traffic. The Scope tab would be of interest because that's where you would define the remote ip of the web server(s).
Its probly better to block in the router or if you want it on a machine only do a redirect in the hosts file. en.wikipedia.org/wiki/Hosts_(file)