I have a SuperMicro server with an NVMe boot drive and an additional SSD for data. I was attempting to install ubuntu on the NVMe boot drive and even though the installation succeeds and the correct boot device is set in the BIOS, the server does not seem to be able to boot the newly installed operating system.
I have found a solution to this problem. It turns out that it is not possible to boot from NVMe in BIOS mode. Only UEFI is supported. However, my install disk was booted in BIOS mode, so the installer tried to configure things as such. The solution was to go into the BIOS setup, force the boot order to only consider UEFI devices and then re-install the operating system. The Ubuntu Wiki has more information on making sure to install in UEFI mode. In particular, if the boot menu is black rather than purple, you're in UEFI mode.
Assuming your Supermicro motherboard is intel 60x / x79 based.
Use Clover EFI bootloader or modify the motherboard bios.
The procedures are very well described in this forum:
You might even find a modded bios in the sub-forum.