I've got a connection in a datacenter where the network provider gives me two Ethernet connections. They're supposedly hooked up to the same VLAN, such that I can wire them up to my switch and only one of them will be active at a time, but either side could then do hardware maintenance (rewiring, switch upgrades, etc.) without causing a service outage.
I've partitioned my switch off to have a separate VLAN for this external edge - let's say that ports 1-3 are on the VLAN, with ports 1 & 2 being my colo-provided internet connections, and port 3 being the outside interface of my firewall. This works fine with either port 1 or port 2 connected, but about 2 minutes after both are connected simultaneously, my switch becomes unresponsive, I get about 80% packet loss, and doing some diagnostics show millions of broadcast packets per minute.
I have a basic understanding of STP to know that it should be enabled for this to work; while STP is turned on, both interfaces still get marked as Forwarding.
Anyone have any ideas on what would cause the packet storm? Is there a better way to set up a redundant connection?