Short version - how well do current hypervisors support vGPU with VM suspend/resume capability?

Longer version:

I've used VMware Workstation for many years in my home lab, with the GPU (used for graphics not compute) shared between VMs, and VMs suspended and resumed at will. I'm interested in moving to a bare metal (Type 1) hypervisor, due to limitations in running a hypervisor on top on a full OS. For example, my setup struggles with > about 3 VMs, struggles with memory and resource sharing, isn't quite as stable as a bare metal hypervisor might be, etc.

But getting information on how the various hypervisors work with vGPU is opaque to say the least. I've got use of an early nVidia GRID K2 card, and hoped to use ESXi, but so far this is the sum of my knowledge:

  • GRID/ESXi - My existing K2 is EOL. It'll work up to ESXi 6.5 only, and suspend requires >= ESXi 6.7 (great!). I can use ESXi 6.7 only with newer GRID, but all newer GRID requires commercial cost-level licensing which is totally out of my league. Running unlicensed would be a breach of the EULA and therefore not okay, but even if I did it briefly, for test purposes and evaluation only, the implications seem to shift every few months as new GRID software is released - there's no stability to it. If I could use my K2 I'd be happy, but its compatibility is very limited right now, it seems, and will vanish in a while.

  • Non-ESXi hypervisors - Other reputable and solid hypervisors exist too (KVM, open source + proprietary (Citrix) Xen, bhyve, Hyper-V, QEMU, VirtualBox). But not all are equally robust/efficient at running VMs, and finding with certainty, which of these supports vGPU (on the K2 or any other GPU), and also has suspend/resume capability with vGPU in use, is incredibly tricky.

    Its also usually unclear up front exactly what's supported, what incurs extra licensing fees, etc. (For example, finding out how much is possible with Hyper-V and RemoteFX without a separate remotefx/RDS license). Or perhaps, if I use a different type 2 hypervisor, I will find I don't need a type 1 hypervisor to get around my current setup headaches.

  • Other vGPUs - There are also other vGPU providers. Intel has its Iris Pro, AMD has a few too. Iris Pro has advantages (cost built into CPU, likely won't have support for older versions withdrawn quickly), and AMD seems costly. But again, Iris Pro is fairly newish and finding which hypervisors can/will be able to utilise it and allow suspend/resume is not easy at all. Also Iris Pro comes at a cost of CPU cores, which is exactly what you don't want for a hypervisor, and isn't available on Xeon either (which have more cores) AFAIK.

  • RemoteFX - RemoteFX is a question mark all on its own. I use VMs for (mainly) Windows and (sometimes) BSD desktops, and rarely, family use them for very light Windows gaming, so Hyper-V + Gen1 VM is interesting. But figuring RemoteFX support/leveraging of vGPU isn't easy, in addition to finding about suspend/resume, plus perhaps options for other transports (Teridici cards?). And of course, RemoteFX is being retired and we have little info on how its successor will work. So clarity about the state of play is really hard to find.

I'm not looking for a recommendation here.

I'm just trying to understand more clearly, how things actually stand - which mature/robust hypervisors both support vGPU and also support suspend/resume of vGPU based VMs.

And also any significant conditions/restrictions/limitations which would apply, if any? (Number/size of displays and VMs, accelerates some things not others, etc)

Thanks for any help in clarifying this!

Ideally I'd like to know the GPU families they can do vGPU with, or any other major limitations/requirements - I imagine "Windows guests only" will be a common one and that's ok. But that's a bonus, compared to the main point.

  • 1
    Half hate to downvote it, but it IS a product recommendation and "just try to understand" is a book of product reviews and roadmaps, and thus waaaaaay too broad.
    – TomTom
    Jan 28, 2019 at 19:17
  • I'm not after a recommendation, which would be "I suggest product X". I really want to do my own research, but this one aspect is very opaque, and I am looking for clarity, which hypervisors have what capabilities. I might eventually choose one, but that's separate, and not just based on this answer, by a long way.
    – Stilez
    Jan 28, 2019 at 19:23
  • Yeah, but then this part is WAAAAAYY too broad for a q&a site.
    – TomTom
    Jan 28, 2019 at 19:24
  • I agree. Also, any suggestion we can give would likely become obsolete in a few months. This is one of those questions, with its combination of "which hardware can do that with which software at this point in time".
    – Massimo
    Jan 28, 2019 at 19:38
  • Maybe its coming over as asking for more than intended? A sufficient answer would be something like - "Hypervisors that support vGPU suspend are - esxi version 6.7 onwards using <list> GPUs, kvm v.XX onward using <list> GPUs, xxx3 doesn't support it yet (under development), Hyoer-V supports from v.XX onward but only OpenGL XX and DirectX XX. Screen size is limited in XX and number of displays is limited in YY." A simple list of support availability to.clarify the position for the major hypervisors Ive named, would be a perfect and good answer.
    – Stilez
    Jan 28, 2019 at 19:40


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.