3

Here's a curious ones for the gurus:

Setup:

Source Machine: Windows Server 2003 R2 machine with local hard drive. VHD file of 40GB. 1 x 1Gbps network card, Cat6 cable, switch.

Target Machine: Windows Server 2008 R2 machine with iSCSI connection to iSCSI target on separate machine (1TB, RAID5). 1 x 1Gbps network card, Cat6 cable, connected to same switch as for Source Machine. Second 1Gbps network card, Cat6 cable, connected via isolated switch to the iSCSI target.

Switches are Netgear JGS524 model (web managed).

If I copy from the Win2003R2 machine to Win2008R2 machine local drive I get 40GB in 45 minutes, 36 seconds.

If I copy from the Win2008R2 machine to the iSCSI target (local drive to iSCSI target) I get 40GB in 37 minutes 56 seconds.

If I copy from the Win2003R2 machine to the iSCSI target via the Win2008R2 machine I get 40GB in 3 hours, 50 minutes, 24 seconds.

All copies were done via the following command issued on the Win2008R2 box:

XCOPY <source> <target> /J

XCOPY /J - Copies using unbuffered I/O. Recommended for very large files.

So, what's the bit I'm missing here? Why does a back-to-back copy take in total 1 hour, 23 minutes, 32 seconds when a "straight through" copy take almost 3 times as long?

Switches show no errors, network hovers around the 3% utilisation mark for the duration of the copy (whereas the "back-to-back" copies are around the 25% utilisation mark).

What have I missed?

Improve this question
5
  • Are you using Jumbo Packets? Not that it should matter that much, but it will drastically reduce the amount of processing required by the 2k8R2 machine to forward the data to the iSCSI
    – Mark Henderson
    Feb 1, 2010 at 4:00
  • No, Jumbo Frames are not in use. We initially saw this with Jumbo Frames turned on and thought that was the issue. So, we disabled them (restarted all, including the switch), brought it back to "plain vanilla" TCP and are still seeing this bizarre behaviour.
    – Rick
    Feb 1, 2010 at 4:43
  • Have you got hardware flow control enabled on the switch and server NICs? And are you 100% certain you're running GigE Full Duplex on both ends of the W2K8 box, you seem to be given the standard copy performance but it's worth checking.
    – Helvick
    Feb 1, 2010 at 8:14
  • I have just checked both - yes, hardware flow control enabled across all devices and also full duplex. What's curious is that A+B <> C (ie. if I copy from 2k3r2 to local + local to iSCSI it's way faster than 2k3r2 to iSCSI).
    – Rick
    Feb 1, 2010 at 8:48
  • How are the NICs in the 2008R2 machine arranged? If they're both on a PCI bus, you'll have a performance hit simply because the bus isn't fast enough to handle the traffic. More generally, 40GB in 38 minutes is miserable for gigabit, it's only 18MB/sec, which isn't even twice as fast as 100BaseT. Either you're using very slow drives, or there's a performance problem you've not identified yet somewhere in the system.
    – xenny
    Apr 24, 2010 at 16:24

3 Answers 3

Reset to default
1

Could it be the 'unbuffered' copy the problem? It is possible that Windows does some tricks that can speed up the copy the the source/target is a local disk, but it reverts to a safer behaviour if it is using two net devices.

I have played with disk testing in Unix, and the OSes can play lots of tricks with the disk subsystem. Good luck.

Improve this answer
1
  • Could be. We deliberately used the /J switch to remove any buffering - but we may have tripped over some other "undocumented" feature with this setting. When we didn't use the /J the copies actually took longer - but I don't have the actual figures handy on this.
    – Rick
    Feb 2, 2010 at 22:55
0

What about the SMB-Protocol? Win2k8R2 uses SMB 2.0 while older Win-versions only have SMB 1.0 which is not as fast AND is there a virus-scanner active? On the other way the direct access of the iSCSI-Device uses different protocol with minimized overhead and no virus-scanner for shure.

Improve this answer
0

Firstly, you are copying from 2003R2 to 2008R2 in both instances. Since 2003 is involved it can only use SMB1 and doesn't do much in the way of simultaneous requests and the information will be traversing the network in chunks of around 64kb and each 64kb chunk has to be acknowledged by the server as being written before the 2003R2 box sends the next one.

Now, if the 2008R2 box has to send off the iSCSI request and receive an acknowledgment before it returns the reply to the 2003R2 box then this can slow down the process. Some back of the envelope calculations suggest that for 64kb chunks you'd need 22ms between the request from 2003R2 to write a chunk and the response to say that it has been written. That seems a bit long, but is not outside the realm of possibility with all the steps involved.

I'm not sure if that is your problem, but if you are interested then you can use wireshark to look at the network traffic and verify what block sizes and delays are involved.

Another less exciting possibility is that you have your server set as Full Duplex and your switch set as auto negotiate. This combination doesn't work and results in the switch thinking the connection is half-duplex and causes it to occasionally drop packets. The dropped packets would be much worse when the server is both sending and receiving large amounts of data at the same time as would be the case with your second copy process.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .