As others have said, it really depends on how fragmented your data is. The easiest and cheapest route is to just use the disk management tools that come with Windows to see the defragmentation levels, and then make a decision from there.
Host Issues: The host obviously has the OS files and the virtual machine hard drive images (assuming they are physical files locally - other storage technologies will be different). They will benefit from defragging if they are fragmented, but typically they don't fragment too badly. The worst offenders will be the dynamic drives that change size as the data they contain increases. There are tools you can run to compact them down, but really for a dedicated server you are probably better off just converting them to fixed size images, defrag them once and then forget about it.
Guest Issues: Fragmentation issues can still be a problem, so it is a good idea to at least check each guest to see what fragmentation exists (again, using the available tools). For dynamic disks, running a defrag inside the guest should really be accompanied by zero-ing out the available sectors in the virtual disk so that it can be reclaimed and optimized. Again, if you have the available storage to just convert them to fixed size drives, I would really recommend that. I typically use sdelete.exe from Sysinternals like this:
sdelete -c c:
For Linux I would imagine you could use /dev/null or something to fill in the blanks.