Virtual machines in every hypervisor that I know of are able to use additional virtual CPUs on which to schedule additional concurrent threads of execution.
It's exactly that layer of abstraction between physical machine and virtual machine that makes it not work the way you describe. The VM is not aware of how many cores the physical machine has. The VM does not "see" the physical CPUs (or cores) on the physical machine. The hypervisor gives the VM how ever many virtual CPUs, and the guest OS uses those virtual CPUs to schedule additional concurrent threads... The total number of virtual CPUs the hypervisor hands out to the virtual machines can even exceed the number of physical CPUs/cores in the machine.
Said another way, a virtual machine, when assigned a single vCPU, schedules its threads as if it only had one CPU. It doesn't matter how many cores are in the underlying physical machine. (Though it is worth noting that the physical machine may schedule that one VM thread on one physical core for one thread quantum, or time slice, and then run it on a different physical core the next time it's scheduled to run. The virtual machine has no idea any of that is happening though. All it knows is that it can only schedule one thread at a time, one after the other, because it only has one virtual CPU.)
And let's be very clear about our terms here. You assign vCPUs, or virtual CPUs, to VMs, not "cores". Cores (by which I assume you mean physical processing units that share a single physical socket) do not equal vCPUs. There is a layer of abstraction between them. If a VM only has 1 vCPU assigned to it, it can only schedule one thread to run at a time. That is why your VM runs faster with 2 -4 virtual CPUs assigned to it - because it is now able to schedule more than one thread to run concurrently.
However, there is definitely a law of diminishing returns here, as an excessive number of virtual CPUs incur a higher and higher overhead cost in things like synchronization, etc.
There are slight differences between how Hyper-V and VMware hypervisors schedule virtual machine threads for execution, and they differ in their approach to physical resource "oversubscription," but this is a good general concept to start with.