I'm trying to setup a specific VMs cluster using vagrant on my local notebook. I have a Win10 Pro with HyperV as a host. Unfortunately I cannot change it thus I want to create a Linux VM on HyperV (I called it 'nested') with nested virtualization enabled. Then on 'nested' Linux I want to create several VMs using KVM/libvirt. For simplicity in an example code there is only one ('guest').

The assumption is to make a minimal setup on Windows Host and to provision everything by Vagrant. But in fact it's not so easy because HyperV provider does not allow to setup a static IP in both public and private interfaces.

The goal is to create a private network (e.g. 10.75.10.0/24) and to assign each VM a static IP address and allow to see each other. I partially achieved that. My VMs can ping each other except the nested one ('guest1'<->'guest2', 'guest'<->Host, Host<->Nested). guest<->nested don't see each other in both directions.

Requirements: 1. Win10 with HyperV enabled 2. User needs to be in HyperV admin group

I created an internal switch in Host using PowerShell script (Adm):

New-VMSwitch –SwitchName “NAT-Switch” –SwitchType Internal –Verbose
Get-NetAdapter
#put the proper index in the next line
New-NetIPAddress –IPAddress 10.75.0.1 -PrefixLength 24 -InterfaceIndex 16 –Verbose
New-NetNat –Name NATNetwork –InternalIPInterfaceAddressPrefix 10.75.0.0/24 –Verbose
Get-VM | Get-VMNetworkAdapter | Connect-VMNetworkAdapter –SwitchName “NAT-Switch”

ipconfig:

Ethernet adapter vEthernet (NAT-Switch):

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter #5
   Physical Address. . . . . . . . . : XXXX
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : XXXX
   IPv4 Address. . . . . . . . . . . : 10.75.10.1(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
   DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
                                       fec0:0:0:ffff::2%1
                                       fec0:0:0:ffff::3%1
   NetBIOS over Tcpip. . . . . . . . : Enabled

This is my project structure:

./nested
./nested/Vagrantfile
./provision.sh
./Vagrantfile

cat Vagrantfile

$bridge_script = <<EOF
    ifconfig eth0 10.75.10.10 netmask 255.255.255.0 up
    route add default gw 10.75.10.1
EOF

Vagrant.configure("2") do |config|

    config.vm.box = "generic/ubuntu1804"
    config.vm.define "nested"
    config.vm.hostname = "nested"
    config.vm.provider "hyperv" do |hv|
        hv.cpus = "2"
        hv.memory = "2048"
        hv.maxmemory = "2048"
        hv.enable_virtualization_extensions = true
        hv.differencing_disk = true
    end

    config.vm.network "public_network", :bridge => "NAT-Switch", auto_config: false
    config.vm.provision :"shell", run: "always", inline: $bridge_script
    config.vm.provision :file, source: "nested", destination: "~/"
    config.vm.provision :shell, :path => "provision.sh"
    config.vm.provision :shell, inline: "sudo -i -u vagrant bash -c 'cd ~vagrant/nested; vagrant up --provider=libvirt'"

end

cat provision.sh

apt-get update
apt-get install -y bridge-utils qemu-kvm virtinst libvirt-bin qemu-utils qemu ebtables dnsmasq vagrant vagrant-libvirt
usermod -aG libvirt-qemu,libvirt vagrant
systemctl enable libvirt-bin
systemctl start libvirt-bin
vagrant plugin install vagrant-libvirt
vagrant plugin list

cat nested/Vagrantfile

Vagrant.configure("2") do |config|
        config.vm.box = "generic/ubuntu1604"
        config.vm.define "guest"
        config.vm.network "public_network", ip: "10.75.10.11", netmask: "24"
        config.vm.hostname = "guest"

        config.vm.provider :libvirt do |libvirt|
                libvirt.uri = 'qemu+unix:///system'
                libvirt.cpus = "1"
                libvirt.memory = "1024"
        end
end

After running:

vagrant --provider=hyperv

I get the following result:

nested VM:

vagrant@nested:~$ ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:15:5d:38:01:5c brd ff:ff:ff:ff:ff:ff
    inet 10.75.10.10/24 brd 10.75.10.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::215:5dff:fe38:15c/64 scope link
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:6f:8e:84 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:6f:8e:84 brd ff:ff:ff:ff:ff:ff
5: virbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:6c:f9:2f brd ff:ff:ff:ff:ff:ff
    inet 192.168.121.1/24 brd 192.168.121.255 scope global virbr1
       valid_lft forever preferred_lft forever
6: virbr1-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr1 state DOWN group default qlen 1000
    link/ether 52:54:00:6c:f9:2f brd ff:ff:ff:ff:ff:ff
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:bc:01:c9 brd ff:ff:ff:ff:ff:ff
8: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/ether 52:54:00:a7:eb:55 brd ff:ff:ff:ff:ff:ff

vagrant@nested:~$ route -n

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.75.10.1      0.0.0.0         UG    0      0        0 eth0
10.75.10.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.121.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr1
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

For VM1:

vagrant@guest:~$ ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:bc:01:c9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.121.45/24 brd 192.168.121.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:a7:eb:55 brd ff:ff:ff:ff:ff:ff
    inet 10.75.10.11/24 brd 10.75.10.255 scope global eth1
       valid_lft forever preferred_lft forever

vagrant@guest:~$ route -n

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.121.1   0.0.0.0         UG    0      0        0 eth0
10.75.10.0      0.0.0.0         255.255.255.0   U     0      0        0 eth1
192.168.121.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0

I hope someone can recreate the state. I provided a 99% automated scripts.

The question is: How can I improve this setup to make nested vm visable? Manipulating with interface addressing and routing cause a long break for vagrant to up. Can I also improve that? Is there a better way to setup network in nested vm?

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.