Short version - how well do current hypervisors support vGPU with VM suspend/resume capability?
I've used VMware Workstation for many years in my home lab, with the GPU (used for graphics not compute) shared between VMs, and VMs suspended and resumed at will. I'm interested in moving to a bare metal (Type 1) hypervisor, due to limitations in running a hypervisor on top on a full OS. For example, my setup struggles with > about 3 VMs, struggles with memory and resource sharing, isn't quite as stable as a bare metal hypervisor might be, etc.
But getting information on how the various hypervisors work with vGPU is opaque to say the least. I've got use of an early nVidia GRID K2 card, and hoped to use ESXi, but so far this is the sum of my knowledge:
GRID/ESXi - My existing K2 is EOL. It'll work up to ESXi 6.5 only, and suspend requires >= ESXi 6.7 (great!). I can use ESXi 6.7 only with newer GRID, but all newer GRID requires commercial cost-level licensing which is totally out of my league. Running unlicensed would be a breach of the EULA and therefore not okay, but even if I did it briefly, for test purposes and evaluation only, the implications seem to shift every few months as new GRID software is released - there's no stability to it. If I could use my K2 I'd be happy, but its compatibility is very limited right now, it seems, and will vanish in a while.
Non-ESXi hypervisors - Other reputable and solid hypervisors exist too (KVM, open source + proprietary (Citrix) Xen, bhyve, Hyper-V, QEMU, VirtualBox). But not all are equally robust/efficient at running VMs, and finding with certainty, which of these supports vGPU (on the K2 or any other GPU), and also has suspend/resume capability with vGPU in use, is incredibly tricky.
Its also usually unclear up front exactly what's supported, what incurs extra licensing fees, etc. (For example, finding out how much is possible with Hyper-V and RemoteFX without a separate remotefx/RDS license). Or perhaps, if I use a different type 2 hypervisor, I will find I don't need a type 1 hypervisor to get around my current setup headaches.
Other vGPUs - There are also other vGPU providers. Intel has its Iris Pro, AMD has a few too. Iris Pro has advantages (cost built into CPU, likely won't have support for older versions withdrawn quickly), and AMD seems costly. But again, Iris Pro is fairly newish and finding which hypervisors can/will be able to utilise it and allow suspend/resume is not easy at all. Also Iris Pro comes at a cost of CPU cores, which is exactly what you don't want for a hypervisor, and isn't available on Xeon either (which have more cores) AFAIK.
RemoteFX - RemoteFX is a question mark all on its own. I use VMs for (mainly) Windows and (sometimes) BSD desktops, and rarely, family use them for very light Windows gaming, so Hyper-V + Gen1 VM is interesting. But figuring RemoteFX support/leveraging of vGPU isn't easy, in addition to finding about suspend/resume, plus perhaps options for other transports (Teridici cards?). And of course, RemoteFX is being retired and we have little info on how its successor will work. So clarity about the state of play is really hard to find.
I'm not looking for a recommendation here.
I'm just trying to understand more clearly, how things actually stand - which mature/robust hypervisors both support vGPU and also support suspend/resume of vGPU based VMs.
And also any significant conditions/restrictions/limitations which would apply, if any? (Number/size of displays and VMs, accelerates some things not others, etc)
Thanks for any help in clarifying this!
Ideally I'd like to know the GPU families they can do vGPU with, or any other major limitations/requirements - I imagine "Windows guests only" will be a common one and that's ok. But that's a bonus, compared to the main point.