Some ideas for you:
I did a job like this for a Customer a few years ago. I configured Windows XP to start Internet Explorer in "Kiosk Mode" as the shell, and disabled the context menus. No keyboard was present on this unit, so the user couldn't exit full screen mode.
If I had to do this today, I'd start looking around for kiosk extensions for Firefox or "shells" for Internet Explorer as a first step.
To prevent memory leaks in the browser from taking the unit down I used a screensaver to logoff the computer after an inactivity timeout, whereupon it auto logged-on again. I also scheduled a nightly reboot to keep the background processes from leaking away memory.
On the recovery front, it would be fairly trivial to build a Windows PE / BartPE bootable USB stick to re-image the machine (ImageX, Ghost, whatever you want to use) if it failed.
On the "access only certain web site" front: You might be able to get away with pointing the DNS on the PC to a restrictive DNS server that only served the zones for sites you wanted to enable access to. You could use a "HOSTS" file, but then you run into update issues if any of those IPs change. You could use some "netnanny" type filtering software on the PC, but I have no experience with that to speak to. Finally, you could configure the PC to use a proxy server (either hosted locally on the PC, or centrally on the 'net) and filter requests that way.
That old kiosk job was really fun, in part, because the kiosk itself had no Internet connectivity! It ran an Apache / PHP / MySQL stack and a wildcard DNS server in the background and served up the site to itself and the other kiosks in the area via WiFi! You could even load updates to the content on the kiosk from USB memory stick or CD (with a proper authentication file on the media). If you tried to surf away from the main site it was hosting, a wildcard virtual host and the wildcard DNS would take you to a "Sorry, you can't get there..." page.
I would've liked to have used a Linux-based solution for that job, but the Customer had a requirement that I use Windows. I would've ended up doing a very similar thing w/ a Linux-based system anyway, since it allowed the Customer to load their live web-site onto a kiosk that otherwise had no Internet access and didn't require their web site developers to make any changes (i.e. the kiosk executed all of the PHP and used the database in the same way as the real web site). It was loads of fun!