Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

we are currently experiencing very slow speeds when copying a huge file (35GB+), to our surprise it starts off at a very high transfer rate 300MB/s, then drops to 4MB/s after 10-20 seconds into the transfer. the file is being transferred from the iSCSI LUNs to the local drives on the machine, also tried to copy the file to /dev/null results are the same, there is a 1Gbs link going into the target, target, switch and initiator are all on a private VLan, the MTUs are also set at 9000 on all three components.

can anyone help us understand why the transfer suddenly goes down to 4MB/s?

thank you for your time.

Storage is NETAPP FAS2040, RHL 5.2, CISCO Switches.

Regards,

share|improve this question
    
taking into consideration the suggestion thus far given, all jumbo frames have been removed, from switch, storage a host, all NICs reconfigured to 1500 mtu, unfortunaly the problems still remains, could this be related to software? or the iscsi configuration on the host? also note that there is no multipath configured on the host. –  Vlad Dec 23 '13 at 21:21

2 Answers 2

Your switch is likely unable to handle the 9000 MTU so it's having to re-transmit a lot of packets. Change your MTU to the default 1500 and see if the speed improves.

Other factors include host load (doesn't matter if /dev/null or not, network adapter can still be loaded) and the maximum rate of the storage's drives. Since it's at ~4MB/s, I doubt it's a drive bottleneck but your router discarding tons of packets due to too high MTU.

share|improve this answer

If you have decent cisco switches and MTU of 9000 probably is supported, but you need to make sure jumbo frames are enabled everywhere in the path of you will see packet re-transmits.

To test your path support jumbo frames is pretty easy. From your host, ping your netapp with a packet size of 9000. You'll want to ping from both interfaces to both target addresses (assuming you're mutli-pathing). Finally, that's not a 100% guarentee, next you need to look at your switch to confirm there's no fragmented packets. I think there's a counter, and if so, you shouldn't see any packets.

With all that being said, i doubt its jumbo frames, and more so, jumbo frames is probably netting you very little performance wise over a 1g link. If you have dual 1g links, at best you should hope to see 200 - 240MBps. I'm not sure where 300 came from, mathmatically speaking that sounds close to impossible. However, 4MBps copying a 35GB file, sounds very low. That's a large file, which should be 100% sequential IO. I suspect its not related to your SAN or your switching and more related to your host. What is the disk like that you're copying too? How many disks, what raid if any? Any BBC? Have you tried a different host? Also, how is your multi-pathing setup? Are you using round robin? I suspect you are if you hit over 120MBps.

share|improve this answer
    
I've seen download counters vastly over-calculate speeds for the first few seconds of transfer. It's usually a sign of packet loss as the program can't figure out exactly how fast the packets are going. –  Nathan C Dec 23 '13 at 19:28
    
perhaps, but its fairly easy to tell if you're having issues with jumbo frames. Most switches do not fragment packets by default, so if you ping with a 9000 byte size, it should either fail or succeed. if it succeeds, its easy enough to look at the counters and confirm that the switch is fragmenting the packets or if they're coming across as jumbo. Jumbo frames is one of many areas to look into, but i wouldn't focus solely on it. –  Eric C. Singer Jan 1 at 23:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.