There isn't a quick and easy way to set processor affinity for VMs in Linux that I'm aware of, having used VMWare and VirtualBox both, but either should work fine. If you're really keen on speed, you may want to go with KVM completely, if only for the native integration vs. a few layers added for VMWare/VirtualBox. Sounds like you really don't need much for gfx, sound, etc. so that might be optimal for you.
The next part of this answer is a bit off-topic as you asked for a full VM solution, but I'm posting anyways because it may make more sense in your situation. Using a VM sounds awfully heavy-weight, when you could use a container instead to get the same benefits (isolation) without the drawbacks (VM overhead). Yes you will have your environment jump from processor to processor over time, but then again all of the VM solutions will attempt that out of the box, because they are tied to the system scheduler, and the scheduler is what is causing this to happen.