I am trying to learn Datacenter networking and I am finding it hard to understand best practices for IP Address and VLAN assignment to servers and racks.
For my question, I'll assume a typical 3 tier network with Top of the Rack (TOR) switches acting as the access switches. I want to understand what things i'd consider to assign IP addresses and VLANs to the servers in the racks (and across racks). Lets assume IPs are assigned statically. Further, lets assume 10 racks of 40 servers each (so 400 servers in total)
I understand that there are a million ways to design the network but I am really just looking for best practices (or pointers to them).
One way to assign IPs is to assign one big subnet (/16 or a /24 depending on number of hosts) to the entire datacenter. I could then daisy chain the ToRs with VLAN trunks. From what i've read, i could create VLANs of type Management, vMotion/live Migration, application data. However would this require a server to have 3 NICs?
Second way I could think of to do this is to assign IP addresses to 1-400 servers in sequence, but NOT connect the TORs directly (but connect them through a router). I am confused on what behaviour this would achieve. All servers will be on the same subnet since they have same network ID. However since switches are connected only via routers, would all these servers still be on same L2 domain?
Third way is to assign a subnet to each rack but that would waste a lot of IP space.
Any guidance on which way is best practice? ALso, in each of the ways, how should I think about using VLANs along with subnets?