Backing up would essentially mean shut down your VMs and copy all the files from the VMFS to somewhere else. Copying everything would include the snapshots.
If this is a standalone host
You need to backup and restore anyway because if you want to actually upgrade the VMFS you can't do this without creating a new one, meaning trashing the old one if you don't have space somewhere else.
If these are a few servers with shared storage
In order to upgrade to ESX5 you would first upgrade vCenter or install a new one and reattach the servers to it. A newer vCenter can talk to older and newer ESX versions.
Then you vMotion the VMs away from the host to be upgraded, and frankly I would simply reinstall that host with the new version and not play with Update Manager or the like.
You need not immediately convert the datastores nor do you need to upgrade the VM Hardware rightaway. Those would be the steps once you know that the new ESX version works fine for you.
Then you would
- create new datastores for VMFS5 (upgrading to VMFS5 does not really make them become a full-fledged VMFS5)
- storage vmotion or copy the old data to them
- delete the VMFS3s and create new VMFS5s in their place
Finally you will at some time need to
- upgrade VMware Tools
- the VM Hardware version
but these will require shutting down/restarting the VMs.
In summary, the kind of downtime you need depends on the amount of hardware you have (one or more servers) and your VMware licenses (multiple ESX hosts, vCenter, Storage vMotion, ...).