As already said by Bart VirtualBox is more oriented to a desktop kind of usage. There it has the bonus of being truly multi platform (you can get VB for Linux, OSX, Solaris, Win) and having a nice interface.
For simple server installations I'm more used to run VMware Server, I find it is more suitable. My host system of choice is an Ubuntu LTS, and I have a range of systems from old Ubuntu 6.06 + VMwareServer1 to Ubuntu 10.04 + VMwareServer2.
VMware Server already includes the infrastructure to autostart VMs at boot (VirtualBox does not), including options to start one at a time (to not overload the host).
Also, I found VMware Server more suitable to run exotic things (last week we succesfully ported a SCO OpenServer 5 system from metal to VMware Server 2 - while it did not even boot in VirtualBox during a quick test).
VMware Server 2 includes a nice WEB admin interface (useful if your server is headless), but it lacks a decent console (its own native one is a browser plugin for linux and windows, no Mac). Luckily you can just edit VM properties to add a VNC console and use that :)