So the scenario is we have a web server at our colocation. In front of that web server are two SonicWalls and two L2 non-stacking managed switches. We're trying to find an efficient way to set up redundancy on all levels here (router, switch, machine) with the equipment we already own (so please, no "Go buy X" answers). The SonicWalls have an active/passive HA link, the switches have nothing, and the machine has dual NICs teamed with the same IP address (different MACs).
So each router has a link to each switch. The switches have a trunk link between each other and each switch has a connection to the dual NIC on the PC. Right now, the redundancy works such that I can kill the link between the router/switch or the switch/pc and I still have network access. To accomplish this, I set the ARP flush to 2 minutes on the Sonicwalls (lowest it would go) and 60 seconds on the switches). I can set up a constant PING on the machine and then remove one of the links. I have usually 20-30 seconds down time and then the ping starts back up.
My question is, what are the negative effects of having this low arp flush time? The server runs an online testing platform that people use, and in some cases they'll be downloading audio files. I was told by a peer that if they're downloading a file and the arp cache is flushed on the router, they'll lose their download. My understanding with TCP was that if it doesn't get an ACK in response, it will keep resending that same packet until it does? Given this information, can you see any issues that I might run into?