You mentioned already ssh, vim and wget which is essential and perfect. Some additional tools that can make life easier:
1. GNU Screen
"GNU Screen is a free terminal multiplexer that allows a user to access multiple separate terminal sessions inside a single terminal window or remote terminal session. It is useful for dealing with multiple programs from the command line, and for separating programs from the shell that started the program." (From the GNU_Screen page on wikipedia)
A main advantage is that you can have one or several virtual terminals that are in the exactly same state as you left them when you come back (i.e. relogin via ssh). This is also good when your connection is broken for some reason.
Screen works indepently from the software you use to connect to the box (it lives on the server), so it combines well with putty or most other terminal software.
This article shows some nice things you can do with it: http://www.pastacode.de/extending-gnu-screen-adding-a-taskbar/en/
2. Midnight Commander
A console based graphical-like browsing tool for viewing and manipulation of files and directories. Can also do secure remote transfers.
For fast, secure and reliable file transfer and synchronisation between different locations
Use of a distributed version control system like bazaar, mercurial or git to update code. Github or Bitbucket offer commecrcial code hosting, but it is not necessary, you can also use it efficiently on your own machines.
Joseph Kern: can you elaborate how you exactly use git for remote config organisation?
5. Terminal Clients
On unix-like-systems they are already on board, on Windows you can use Putty, Tera Term, Mind Term or Pandora. Or make a cygwin installation and ssh from the cygwin terminel windows to the remote boxes (which has more advantages but this is a question of what you prefer).
6. Tunneling and Port Forwarding
It can be helpful to forward certain ports securely to your local machine. For example you could forward the mysql port TCP 3306 or postgres TCP 5432 and install some database administration tool locally.
You can build tunnels from Windwos machines with putty (or command line based with it's little brother plink), with cygwin and Mindterm also can do port forwarding. If you are locally on a unix-like machine you can use ssh odr plink to create such tunneling.
To create some more stable and permanent tunneling for various ports I recommend OpenVPN. The "pre-shared-key" tunneling method from point to point is not so hard to install.
7. Have a local unix-like system
When your local machine is a Mac you have this already, you can open a local shell. When your workstation is windows-based it could be helpful to create a local unix-like server, which is in the same local network. This can be a different machine in a different room connected to the same router or switch. Or if you want only one machine, you can install the free vmware server and make a virtual machine, preferably the same operating system as your remote machine. Install a samba server on it and you can "net use" the samba shares from your desktop.
If you an ssh server on the local server and open port 22 on your router for it you can ssh into your local system when you are outside.
You can build tunnels to remote machines or transfer and synchronise files and whole file trees with rsync. You can use it for testing, for VSC, for local development, as a local webserver, for training purposes.
You can pull backups from remote machines. You can create local cron jobs that do te backups autmatically (e.g. databases you want to save locally regularly)