Sounds like you're after a bit of a breakdown about what VMWare products do what.
VMWare ESXi - This is the latest version of the VMWare HyperVisor. It is a dedicated operating system that installs on 'bare metal' - that is, it sits on its own on the box, and you create your VM's inside it. ESXi does come in a free version, which is really quite good, but if you want access to any of the advanced features you will need to purchase a license.
VMWare ESX - This is the previous generation of the VMWare Hypervisor. Seeing as this is a new installation, avoid it.
VMWare Server - This is VMWare software that sits ontop of an existing operating system. To use this you still need to have Windows or Linux installed on the machine, and then install VMWare Server onto it.
VMWare Workstation - this is like VMWare Server, but designed for daily-use desktops, usually for development or testing purposes.
vSphere - this is the management tool used for managing multiple VMWare hosts. The free version of ESXi does come with a free version of vSphere, which is suitable for just the one installation, but as soon as you start to add multiple physical machines you really need a dedicated vSphere installation and management station. This is pretty expensive, but for a single server not required.
There are also free versions of the same software above, but from Microsoft. Look into Hyper-V (there's heaps of questions about it on here).