Is my VLAN configuration ideal for 2 subnets where one contains a hypervisor?

Question

I’m not sure my VLAN is setup correctly or effectively. I’ve had my layer 3 switch for ages (Extreme Summit x450e-48p). Honestly I’ve just plugged everything into it and used it unmanaged. Before the switch is my Cisco ASA 5510 firewall (it’s my gateway, 192.168.200.1). As I said, in the past I just had everything plugged into the switch and never even managed it.

Recently I got a Nutanix Hyper Converged virtual host. Best practices dictated that the host be on it’s own VLAN separate from the broadcast domain. Since I already had the extreme switch (layer 3) I figured with some research I could make 2 VLANs and accomplish this. I logged into the switch and noticed the factory setting includes a single Default vlan (aptly called Default). It was untagged and included all the ports. I needed 9 ports for the Nutanix host so I created a VLAN called “Nutanix” and tagged it as VLAN 10. I added the 9 ports to the Nutanix VLAN. So now I had two VLANs (Default and Nutanix). According to the extreme switch’s documentation, the two vlans needed Intervlan routing configured in order for them to talk to each other, so I enabled this. The vlan ip config is as follows:

Name            VID     Protocol Addr       
---------------------------------------------------------------------------------------------
Default         1       192.168.200.2  /24   (this is where my domain controller resides)
Nutanix         10      10.1.10.2      /24  
---------------------------------------------------------------------------------------------

I setup my DHCP server (that's always been on the default vlan) to serve a scope in the 10.1.10.0 subnet. Once everything was configured and plugged in I connected a laptop to my Nutanix subnet but it would never get an IP address from the dhcp scope. I ended up having use Cisco support to get help with the firewall. They helped me configure a route on the cisco to both VLANs. The lan interface on the cisco (eth0/1) is ‘split’ (lack of a better term). So it’s like its two interfaces, eth0/1 for my Default vlan (ip is 192.168.200.1) and eth0/1.10 for my Nutanix vlan (ip is 10.1.10.1). Heres how it looks on the ASA:

show int ip brief
Ethernet0/1                192.168.200.1  
Ethernet0/1.10             10.1.10.1    

Then we trunked PORT 1 on the extreme switch and connected it to eth0/1 on my cisco. It was trunked for both vlans, tagged on the Nutanix vlan, and untagged on the Default. I really think this is where my problem lies (will explain in a moment). They told me I need to make sure the switch is in Layer 2 mode (I do this by disabling ip routing on the switch) and the cisco will handle the routing.

So my question is, is this the best configuration? It seems like my switch should be a layer 3 and manage routing. I really just want my firewall to protect my network and let things in if they have an access list. My Default vlan clients have to have the cisco’s eth0/1 interface address as a gateway (192.168.200.1) and the Nutanix vlan clients have to have the cisco’s eth0/1.10 interface address (10.1.10.1) as the gateway to get to the internet. Everything appears to work (dhcp, internet, etc…). In my virtual machine host, when I create a virtual machine I have to pick which network I want the machine on and it gets assigned the proper scope. My servers that are Virtual Machines will be on my Default vlan, but the virtual host is actually on my Nutanix vlan.

The main problem I’m having is with some Routing and Remote Access services. I can create a fresh virtual machine, install Server 2012 or 2016 on it, install the Routing and Remote Access role on it, and reboot. When it comes up the Remote Access Management Service will never start and hangs up, which causes me to never be able to configure Routing and Remote Access. I’ve tried several different Server installs and it’s the exact same issue everytime. I think it’s having problems connecting to my Domain Controller thats on the Default Vlan (remember, my virtual machines are on the Default vlan, INSIDE the Virtual Host that is on the Nutanix VLAN). It could be the fact that my Domain Controller is still Server 2008 R2 and the RRAS server is 2016. I’m planning on promoting my domain to 2016 in the future but I feel like the VLAN configuration is going to haunt me in the future. Sorry for the long post. Bottom line question is is this vlan config ideal? I think my switch should be configured for layer 3 and handle routing… firewall should just scan incoming traffic and do its job. I THINK this would mean that my vlan interface ip addresses (192.168.200.2 and 10.1.10.2) would be my gateway for their respective subnet (this could be incorrect).

Edit: An important thing I forgot to include which might help is the fact that Ports 39-44 are all tagged for both VLANS. The reason I think it might be a VLAN issue, is inside my hypervisor there are two networks setup: one tagged for the Nutanix VLAN (tag 10) and one for the Default VLAN (tagged 1). On my cisco, the eth0/1 is NOT tagged for the Default VLAN. This is why I think it would be better if the switch handled all routing and intervlan routing (and DHCP relay for that matter, it's capable of all these). I just don't know how to configure the cisco this way. Is it a matter of just tagging the eth0/1 port to the Default VLAN as well as the Nutanix VLAN?

So sorry for the long post, I appreciate if you've made it this far. Here's a little diagram of the network if it helps:

Answer 1

If you want strong access control between the two VLANs then it makes sense to trunk them to and gateway them on the ASA, and deal with the ACLs there. The way you described doing this (with one tagged VLAN, one native) will work just fine but arguably what Cisco are saying about two sub-interfaces is a little more elegant and scales better for larger numbers of VLANs. This approach (doing all routing on the ASA) also makes routes easier (basically you don't have to configure any except a default route to the internet on the ASA). It does, however, add the ASA as point of failure between the two VLANs as well as the switch.

If you care less about controlling the inter-VLAN routed traffic and more about throughput, then gateway them on your Extreme switch like you describe in the second paragraph. This will make for easier interface config on the ASA but you'll need to configure DHCP relay on the Extreme switch to make DHCP work in the VLAN that doesn't contain the DHCP server (in fact you'd have to do that on the ASA in the method above anyway). If you want to control traffic between the VLANs with this design, you'll have to figure out ACLs on the x450.

Assuming the second approach and that you want the internet to work, then either add a third mini-VLAN between your Extreme switch and the ASA or put the ASA natively into one of your existing VLANs, and add a default route towards it on the switch. Also add a route on the ASA towards the other interior subnets behind the switch so that it can send inbound traffic there. I guess you already have NAT working if internet things have been fine thus far, otherwise you need that too.

As to the Nutanix config, if it expects both VLANs to be tagged/trunked (i.e. one of them is NOT considered native by the Nutanix box) then you need to set this appropriately on the switchport towards the Nutanix. The setup on the ASA doesn't matter; the switch will use the VLANs as tagged and/or native on a per-port basis, it doesn't need to be consistent throughout the whole network.