3

we are currently experiencing very slow speeds when copying a huge file (35GB+), to our surprise it starts off at a very high transfer rate 300MB/s, then drops to 4MB/s after 10-20 seconds into the transfer. the file is being transferred from the iSCSI LUNs to the local drives on the machine, also tried to copy the file to /dev/null results are the same, there is a 1Gbs link going into the target, target, switch and initiator are all on a private VLan, the MTUs are also set at 9000 on all three components.

can anyone help us understand why the transfer suddenly goes down to 4MB/s?

thank you for your time.

Storage is NETAPP FAS2040, RHL 5.2, CISCO Switches.

Regards,

1
  • taking into consideration the suggestion thus far given, all jumbo frames have been removed, from switch, storage a host, all NICs reconfigured to 1500 mtu, unfortunaly the problems still remains, could this be related to software? or the iscsi configuration on the host? also note that there is no multipath configured on the host.
    – Vlad
    Dec 23, 2013 at 21:21

2 Answers 2

1

Your switch is likely unable to handle the 9000 MTU so it's having to re-transmit a lot of packets. Change your MTU to the default 1500 and see if the speed improves.

Other factors include host load (doesn't matter if /dev/null or not, network adapter can still be loaded) and the maximum rate of the storage's drives. Since it's at ~4MB/s, I doubt it's a drive bottleneck but your router discarding tons of packets due to too high MTU.

1

If you have decent cisco switches and MTU of 9000 probably is supported, but you need to make sure jumbo frames are enabled everywhere in the path of you will see packet re-transmits.

To test your path support jumbo frames is pretty easy. From your host, ping your netapp with a packet size of 9000. You'll want to ping from both interfaces to both target addresses (assuming you're mutli-pathing). Finally, that's not a 100% guarentee, next you need to look at your switch to confirm there's no fragmented packets. I think there's a counter, and if so, you shouldn't see any packets.

With all that being said, i doubt its jumbo frames, and more so, jumbo frames is probably netting you very little performance wise over a 1g link. If you have dual 1g links, at best you should hope to see 200 - 240MBps. I'm not sure where 300 came from, mathmatically speaking that sounds close to impossible. However, 4MBps copying a 35GB file, sounds very low. That's a large file, which should be 100% sequential IO. I suspect its not related to your SAN or your switching and more related to your host. What is the disk like that you're copying too? How many disks, what raid if any? Any BBC? Have you tried a different host? Also, how is your multi-pathing setup? Are you using round robin? I suspect you are if you hit over 120MBps.

2
  • I've seen download counters vastly over-calculate speeds for the first few seconds of transfer. It's usually a sign of packet loss as the program can't figure out exactly how fast the packets are going.
    – Nathan C
    Dec 23, 2013 at 19:28
  • 1
    perhaps, but its fairly easy to tell if you're having issues with jumbo frames. Most switches do not fragment packets by default, so if you ping with a 9000 byte size, it should either fail or succeed. if it succeeds, its easy enough to look at the counters and confirm that the switch is fragmenting the packets or if they're coming across as jumbo. Jumbo frames is one of many areas to look into, but i wouldn't focus solely on it. Jan 1, 2014 at 23:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .