You'll get the vast majority of what you want by having the users run with non-privileged accounts. If you let users run with "Administrator" accounts on your terminal server machine you're asking for the box to be destroyed almost immediately. (Besides, users shouldn't even be using their desktop PCs for day-to-day work with "Administrator" rights. The terminal server computer is just a big, multi-headed PC, and no different than the desktop PCs in that respect.)
Beyond that, it looks like you'll want folder redirection to get "My Documents" (and possibly the "Desktop" and "Application Data" folders) onto the server where you want user data stored. You're not really going to be able to stop users from being able to save into the per-user "Temp" directories or under their user profile (w/o breaking how the OS works). Folder redirection and user education are your friends there. Not having "Administrator" rights, though, is going to seriously limit the number of places users can stash files, and make it much more likely that they'll save files in the right place.
Generally, I use a group policy object set in loopback policy processing "Replace" mode, applied to the OU with the terminal server computers in it. (This is a GREAT application for group policy loopback policy processing-- you should read up on it.)
I fill that loopback GPO with all the per-user settings that I want to apply to terminal server users (typically Microsoft Office customizations, folder redirection, tweaks to the Windows look-and-feel, etc).
If you have multiple terminal servers, I'd recommend setting a terminal services roaming user profile for each user (to a location different than their regular Windows roaming user profile) so that their terminal services environment "follows" them between the different terminal server machines.
If you decide that you do want to restrict what programs the user is allowed to run, I suggest you have a look at "Software Restriction Policies" (see http://technet.microsoft.com/en-us/library/bb457006.aspx). You can keep users from being able to execute applications except those that are stored in specific paths (places users aren't allowed to write-- "\Program Files...", "\Windows", etc) or that have specific digital signatures. If somebody downloads an EXE to their %TEMP% directory (a place they're allowed to write), they'll find that Windows won't execute it.