I've never used OpenVZ or Virtuozzo but this is how I understand it:
Burstable RAM is a "feature" of Virtuozzo and its open-source counterpart OpenVZ. On these boxes the individual VPSes don't actually have their own kernels running. Under these technologies each VPS has two limits in the host kernel, privvmpages and oomguarpages.
Privvmpages is set to the total amount of burstable + gauranteed RAM and processes within a VPS are allowed to allocate up to that amount of memory. When the host is running low on RAM it will begin killing processes across all of the VPS that have allocated greater than oomgaurdpages worth of RAM. Remembering that all of the VPSs on the host have all of their processes running within the same kernel makes this a bit easier to understand.
I have no real clue about how the OpenVZ/Virtuozzo OOM killer makes it's decisions about which processes to kill. If I was designing it I'd probably choose the VPS with the most pages greater than oomgaurpages and then choose a process within that VPS based on a score calculated from process age and size, rinse, repeat. Though like I said, I don't know how they really do it.
It is important to note that on a normal Linux system a process can malloc greater than the total amount of swap + RAM that exists without issue. It's when you go to use it that you are stopped. Under OpenVZ/Virtuozzo the malloc will fail. This is actually fairly common in things that would normally have lots of repeated allocations/deallocations when the process wishes to manage it's memory itself. A special purpose memory manager in such a process can have much greater performance.