I need to use Win10 with a server created with VMware ESXi 5.0. I'm going to install Debian 6.0 on VMware ESXi 5.0 and then install VirtualBox 5.0.14 with Win10 guest operating system. Will it work?
This is called "nested" virtualization and it's possible for some combination of hypervisor.
The reason is that for nested virtualization to work, the first hypervisor need to pass to the second one some CPU features (Intel VT-x or AMD-V).
VirtualBox does not support it. It has been a feature request for 7 years now! Source here.
But as I understand it that means it cannot be the first hypervisor, but maybe it could be the second one...
Why not update your ESXi to 5.5 then have directly your Windows 10 on ESXi?
As of November 2018 it seems it will be supported soon.
From the previously linked page:
Nested AMD-V is functional (although not feature complete). Nested VT-x is still work-in-progress.
Update2 - April 2019
From @cursed_axes comment :
With VirtualBox 6.0.6: Oracle VM VirtualBox supports nested virtualization on host systems that run AMD CPUs. This feature enables the passthrough of hardware virtualization functions to the guest VM. That means that you can install a hypervisor, such as Oracle VM VirtualBox, Oracle VM Server or KVM, on an Oracle VM VirtualBox guest. You can then create and run VMs within the guest VM