Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm looking for help with what I'm sure is an age old question. I've found myself in a situation of yearning to understand network throughput more clearly, but I can't seem to find information that makes it "click"

We have a few servers distributed geographically, running various versions of Windows. Assuming we always use one host (a desktop) as the source, when copying data from that host to other servers across the country, we see a high variance in speed. In some cases, we can copy data at 12MB/s consistently, in others, we're seeing 0.8 MB/s. It should be noted, after testing 8 destinations, we always seem to be at either 0.6-0.8MB/s or 11-12 MB/s. In the building we're primarily concerned with, we have an OC-3 connection to our ISP.

I know there are a lot of variables at play, but I guess I was hoping the experts here could help answer a few basic questions to help bolster my understanding.

1.) For older machines, running Windows XP, server 2003, etc, with a 100Mbps Ethernet card and 72 ms typical latency, does 0.8 MB/s sound at all reasonable? Or do you think that slow enough to indicate a problem?

2.) The classic "mathematical fastest speed" of "throughput = TCP window / latency," is, in our case, calculated to 0.8 MB/s (64Kb / 72 ms). My understanding is that is an upper bounds; that you would never expect to reach (due to overhead) let alone surpass that speed. In some cases though, we're seeing speeds of 12.3 MB/s. There are Steelhead accelerators scattered around the network, could those account for such a higher transfer rate?

3.) It's been suggested that the use SMB vs. SMB2 could explain the differences in speed. Indeed, as expected, packet captures show both being used depending on the OS versions in play, as we would expect. I understand what determines SMB2 being used or not, but I'm curious to know what kind of performance gain you can expect with SMB2.

My problem simply seems to be a lack of experience, and more importantly, perspective, in terms of what are and are not reasonable network speeds. Could anyone help impart come context/perspective?

share|improve this question
    
the differences in speed aren't just SMB vs SMB 2. remember that in vista and above the networking stack is no longer a *nix based add on and is integrated into the OS allowing for much lower latency and overhead. –  Jim B Jun 1 '12 at 13:48
    
Thanks Jim, thats good to know. Yeah I know there are a billion variables upstream and downstream depending on where you're looking in the OSI stack. A coworker has offered that perhaps the speed difference could simply be a matter of "SMB vs. SMB2." I guess I'm skeptical of SMB2 (.1) giving us a 10x speed increase, but again, I'm just not versed enough in the aracne arts to know ;) –  John K Jun 1 '12 at 13:52
    
Great question! Any updates on this? And fwiw, a windows XP box that is sharing but not "logged in" transfers files faster than one with a user logged in. –  Chris K Jun 12 '12 at 23:57

3 Answers 3

The mathematical formula you are referring to is actually the way to determine the most efficient transmit window size settings for TCP, not the actual bandwidth available. TCP uses a mechanism called sliding windows that allows for adjustment of transmit speeds based on network conditions. The idea is that a TCP transmitter will send more and more data without requiring an acknowledgement from the receiver. If there's a loss of data then the amount of data sent between acknowledgements decreases, thus also decreasing the effective bandwidth.

The formula in question actually determines the ideal sizing of that TCP transmit window based on the latency and round-trip latency between a given pair of hosts. The idea is to have a window sized such that the amount of data 'in flight' corresponds to what's known as the bandwidth-delay product. For example, if you have 50 megabits per second (6.25 megaBYTES) and an average round-trip latency of 100ms then you'd have 6.25 * 0.1 = 625 kilobytes of data. This would be the value that TCP would negotiate (if configured correctly). As the latency and bandwidth characteristics of your links varies then so too does the window size.

What you need is a bandwidth management tool like iperf (free) running on both the source and your various destinations. This should give you an idea of the actual amount of throughput possible (independent of other apps) while also providing some insight into latency. Running an extended ping between hosts will also provide a general idea of latency characteristics. When you have this data you'll have a better idea of what you should be seeing as far as throughput goes.

BTW - The use of any kind of LAN optimizer will often incorporate data compression, TCP optimization, caching, etc.. While handy, it can obscure the nature of the underlying links. Once you have an idea of the raw bandwidth / delay (and packet loss, potentially) you can take a closer look to make sure your various hosts are set up to take proper advantage of available bandwidth.

share|improve this answer
    
Thanks for the great info rnxrx. The sliding windows used in TCP makes a lot of sense. In our case, since the only traffic we're concerned with is going over TCP, so I guess my question is heavily TCP focused. We have used other methods of data transfer (like repliweb) and we've achieved much higher throughput (up to 11 MB/s). I wasn't aware of the bandwidth delay product though, that's great to know! –  John K Jun 2 '12 at 23:36

Try "ping -l 8092" or FTP or HTTP to check if it is SMB issue.

First of all: what media do you use to connect computers? What us "100mpbs"? Ethernet? You can't use it for "distributed geographically" computers, right?

In case of "vpn over Internet" routers between your computers may use different links: one is fast, other is not. They may choose link based on many parameters.

Please describe your network.

That could be an MTU issue also: several links may have different MTUs.

share|improve this answer
    
Thank llya. I updated the question to answer yours. This is a normal ethernet network, with a 100 Mbps network card in the source host. –  John K Jun 1 '12 at 14:40
    
Josh, I still can not understand:(( Do you connect your PC with Server using 100BASE Ethernet? Even if you use 100BASE-LX10 (ethernet over two optical fibers) you can't connect two computers "across the country": it has distance limit about 10 Kilometers. And in case of Ethernet do you use switch (which?) or connect computers directly? And please tell me about latency: how did you get "72 ms"? how did you calculate it? Thanks –  Ilya Jun 1 '12 at 16:26
    
We're going over a local LAN to the public interenet, to a datacenter and then to the server over a corporate intranet. The latency was determined using ping. We do go through multiple switches as would be expected. The closest switch has a confirmed autonegotiated speed with the host of 100Mbps. Hopefully that helps. –  John K Jun 1 '12 at 16:37
1  
nb - the bandwidth you're actually concerned with isn't the 100M of your LAN connection but the slowest link between the endpoints. This could be your WAN link or some slower intermediate link. –  rnxrx Jun 3 '12 at 1:21
    
That's a great point, I'll update the question, but I know we're hooked up to our ISP via an OC-3. I have to imagine everything upstream of that is at least 100Mbps, and I know the infrastructure in the building is 100M or 1G all the way to that connection. –  John K Jun 4 '12 at 22:03
up vote 1 down vote accepted

A lot of comments and users have offered great advice here. Some of them hit pretty close to what I was looking for, but I was also fortunate enough to meet with a Network veteran from our company that's helped clarify things. I thought I would post my findings/understandings here for the benefit of others. Please feel free to correct me if any of this seems off:

1.) The maximum throughput of a single TCP session with 72ms latency and a 64K window is right around 0.8 MB/s, making that speed reasonable for single thread, single session copies, like the ones we performed with robocopy.

2.) This speed difference appears to come down to the effectiveness of the transfer method. In our case, we were using Robocopy and Repliweb. I discovered robocopy uses a single TCP session, whereas Repliweb can open multiple sessions to send data through.

3.) Research from Microsoft's website does show SMB2 as having a considerable performance gain over SMB1. There have, however, been problems in some cases in how the OS negotiates which protocol to use, so one should be aware of both a.) which cases SMB2 may be used, and b.) whether or not SMB2 is actually being utilized based on network captures.

Currently, it looks like Wire-shark can determine the use of the SMB2 protocol.

I hope this helps. Again, my understanding is fairly rudimentary here, feel free to expand.

share|improve this answer
1  
Aside from the bandwidth-delay calculations, be aware that SMB2 vastly improves performance over a higher-latency link, as it does not wait for the server to acknowledge a write before transferring more data. We do in fact see 10x throughput gains with SMBv2 over our WAN connections using ROBOCOPY. Also consider the /MT thread with robocopy if you're copying lots of files, it does up to 8 in parallel by default. –  rmalayter Jun 5 '12 at 16:28

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.