This would depend on the VPS and what is installed under Ubuntu.
If it's a server install, it probably has a number of GUI (X) items not installed, since it can have a dramatic impact on how much disk space and CPU cycles are used.
If it does have the workstation version with the default workstation packages, there is a desktop sharing ability using VNC to connect to it.
Alternatively, you can get an X server for your Windows system and ssh in with putty (using X-forwarding enabled) to launch graphical applications to appear locally on your Windows machine while running on the remote system.
Your best bet is to learn how to use the command prompt, though. It will let you run a leaner system, lower CPU usage, and simplify how to connect to your machine (just using SSH instead of a slower graphical interface).