When setting up internal cloud, the recommended best practice seems to be to isolate data traffic from management traffic by using separate L2 networks. For example, see the diagram in the OpenStack Quantum documentation).
Note that this management network is used for the various cloud-related services to coordinate. It does not refer to out-of-band management using something like IPMI.
Assuming you have two switches, and two physical NICs on each compute node, this isolation is simple to implement. However, you could also configure the switches for link aggregation (aka bonding, trunking), which would protect against one of the switches failing. In this scenario, you could use VLANs to isolate the networks.
What are the tradeoffs associated with using a physical "data" switch and a "management" switch versus link aggregation + VLANs? In particular, other than the complexity of setting up the switches and the compute hosts to support link aggregation and VLANs, what are the disadvantages of that approach?