Subnet and VLAN addressing in a Datacenter

Question

I am trying to learn Datacenter networking and I am finding it hard to understand best practices for IP Address and VLAN assignment to servers and racks.

For my question, I'll assume a typical 3 tier network with Top of the Rack (TOR) switches acting as the access switches. I want to understand what things i'd consider to assign IP addresses and VLANs to the servers in the racks (and across racks). Lets assume IPs are assigned statically. Further, lets assume 10 racks of 40 servers each (so 400 servers in total)

I understand that there are a million ways to design the network but I am really just looking for best practices (or pointers to them).

One way to assign IPs is to assign one big subnet (/16 or a /24 depending on number of hosts) to the entire datacenter. I could then daisy chain the ToRs with VLAN trunks. From what i've read, i could create VLANs of type Management, vMotion/live Migration, application data. However would this require a server to have 3 NICs?

Second way I could think of to do this is to assign IP addresses to 1-400 servers in sequence, but NOT connect the TORs directly (but connect them through a router). I am confused on what behaviour this would achieve. All servers will be on the same subnet since they have same network ID. However since switches are connected only via routers, would all these servers still be on same L2 domain?

Third way is to assign a subnet to each rack but that would waste a lot of IP space.

Any guidance on which way is best practice? ALso, in each of the ways, how should I think about using VLANs along with subnets?

Answer 1

Not a comprehensive answer, but some things to think about:

The benefit of VLANs is that a NIC can access multiple subnets simultaneously, so in a small or not very busy environment you might find it unnecessary to assign multiple physical uplinks. On the other hand, for all uses where you might see traffic contention (storage, vMotion, VM data) you will most likely want to separate traffic types between physical interfaces.

Of course you will want each connection to provide sufficient redundancy, so that means you need at least two physical interfaces per required uplink.

Regarding static vs dynamic IP addresses, generally speaking when you’re dealing with more than a few hosts you will want to minimize manual work, both for the sake of management overhead, but also to limit the risk of human error. Use DHCP reservations if you need machines to have predictable addresses. Combine DHCP with dynamic DNS to simplify reverse lookups.

If you expect significant traffic between racks you may encounter bottlenecks if you need to route it all through a central point. Network virtualization using for example VMware NSX may help you offload significant chunks of virtual machine traffic from the router to the VM hosts as long as they have a common L2 connection to each other.

If you “only” have dozens of racks, I wouldn’t worry about wasting a /24 IPv4 subnet per rack for management networks: Better safe than sorry, if you ever get the need to increase physical server density.