Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a Windows failover cluster and every time I take a remote robocopy backup from this cluster it seems like it's causing such high network saturation that the cluster monitor thinks the NIC is unresponsive and then fails over.

This is my assumption based on the time of day this usually happens.

I've tried adding the /IPG flag to robocopy to allow some space between packets. Is there anything else I could consider to space traffic out? I've had similar problems that aren't robocopy related. Perhaps a network interface tweak?

Thanks.

share|improve this question
1  
How many NIC's on each cluster host? What services/applications are you clustering? Run the following from PowerShell and post the output here: Get-ClusterNetwork | FT Name, Metric –  joeqwerty Nov 19 '12 at 19:21
    
I don't seem to have that cmdlet available to me. –  eth0 Nov 19 '12 at 22:48
1  
You probably need to import the module: import-module failoverclusters –  joeqwerty Nov 19 '12 at 23:29
add comment

1 Answer

up vote 1 down vote accepted

Take a look at the network during normal hours when you're not running the Robocopy job. Do you see congestion? Do you see a high volume of broadcast traffic? Are links saturated? Take a look at link speed and duplex settings.

In general, take a look at the network as a whole before you start tweaking server and/or switch settings. The key to knowing when something isn't working normally is to know what it looks like when it is working normally. If the network is performing poorly in it's normal state then the added load of your Robocopy job is only going to exacerbate it.

EDIT

In addition, depending on your configuration you can prevent cluster communication (heartbeat traffic) from transiting the network that you're doing the Robocopy job over. If the Robocopy job is running over an iSCSI network then you shouldn't have cluster communication enabled on that network to begin with.

share|improve this answer
    
I have a separate iSCSI switch pair with 4 dedicated interfaces to iSCSI on both servers. I have a separate physical interface on each with a crossover cable for hear-beat. Then I have a pair of teamed interfaces going to my core switch for job traffic. I'll keep a lookout for how much traffic the jobs work up, but I didn't think it would be too much. The jobs run in batch and certain times of the day. Maybe all I need to do is move schedules around. Thank you. –  eth0 Nov 19 '12 at 22:49
    
The cluster will use any network adapter for cluster communications as long as the "allow cluster network communication on this network" setting is enabled, which may not be what you want. I think that knowing the metric for each cluster network is going to go a long way in determining why the cluster is failing over during the Robocopy job. –  joeqwerty Nov 20 '12 at 1:20
    
I see. Here is the output of my network metrics: Heartbeat=1000, iSCSI=10000, LAN=1000 –  eth0 Nov 20 '12 at 3:41
    
I'm fairly new to failover clustering but here's a link that goes into some detail, especially regarding the network metrics - blogs.msdn.com/b/clustering/archive/2011/06/17/10176338.aspx –  joeqwerty Nov 20 '12 at 3:59
    
The Heartbeat and LAN networks have the same metric which means that heartbeat traffic will go over both. If the LAN network becomes saturated and the heartbeat isn't responded to on that network that might explain things. AFAIK, it's recommended to have multiple networks for heartbeat traffic but in your case it may make sense to isolate the heartbeat traffic to just the Heartbeat network (which it seems you were intending with the crossover cable). You can do this by disabling the "allow cluster network communication on this network" setting on the LAN network. –  joeqwerty Nov 20 '12 at 4:03
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.