You should really bump up the RAM on that server, don't forget that each user will need to have the applications they are running at least partially in RAM. Depending on how many users you are going to be connecting, you could easily need 6-8 GB or more of RAM available. Otherwise you are going to see the machine doing a lot of paging to disk and thus thrashing the hard drive(s).
Looking at Windows 2k3 terminal server I have running in a similar setup with 4 users on it currently, using the office suite. The 4GB and 1 Core allocated is just about enough. Of course they are also coming down a very slow pipe so that will limit resource utilization as well, they just can't do things fast.
The R200 can go up to 16GB of RAM, although it's kinda pricey at that point if you can afford it - get it, if not I would go to at least 8GB of ram.