Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a webserver with a current connection of 100Mbit and my provider offers an upgrade to 1Gbit. I understand that this refers to throughput but the larger the packets, the faster they can be transmitted as well, so I would expect a slight decrease in response time (e.g. ping). Did anybody ever benchmarked this?

Example (100mbit to 100mbit server) with 30 byte load:

> ping server -i0.05 -c200 -s30
[...]
200 packets transmitted, 200 received, 0% packet loss, time 9948ms
rtt min/avg/max/mdev = 0.093/0.164/0.960/0.093 ms

Example (100mbit to 100mbit server) with 300 byte load (which is below MTU):

> ping server -i0.05 -c200 -s300
[...]
200 packets transmitted, 200 received, 0% packet loss, time 10037ms
rtt min/avg/max/mdev = 0.235/0.395/0.841/0.078 ms

So from 30 to 300 the avg. latency increaces from 0.164 to 0.395 - I would expect this to be a slower increase for a 1gibt to 1gbit connection. I even would expect 100mbit to 1gbit to be faster, if the connection is through a switch which first waits until it received the whole packet.

Update: Please read the comments to the answers carefully! The connection is not saturated, and I don't think that this speed increase will matter for humans for one request, but it is about many requests which add up (Redis, Database, etc.).

Regarding answer from @LatinSuD:

> ping server -i0.05 -c200 -s1400
200 packets transmitted, 200 received, 0% packet loss, time 9958ms
rtt min/avg/max/mdev = 0.662/0.866/1.557/0.110 ms
share|improve this question

6 Answers 6

The only way latency would drop appreciably is if the current 100Mbit link is saturated. If it is not saturated, you will likely not notice any change.

Additionally, your assumption that the 1Gbit link will be able to support larger packets is incorrect. Max packet size is determined by the MTU of the various devices along the path that the packet takes - starting with the NIC on your server, all the way through to the MTU of your customer's computer. In internal LAN applications (when you have control over all the devices along the path), it is sometimes possible to increase the MTU, but in this situation, you are pretty much stuck with the default MTU of 1500. If you send packets larger than that, they will end up getting fragmented, thereby actually decreasing performance.

share|improve this answer
2  
"Appreciably" is the key word here. I just checked two servers with identical hardware and low network load, but with different ethernet speeds. Average ping time (local, with the ping source on the same subnet) dropped from 0.21 milliseconds to 0.17 milliseconds. Pinging the same servers from home, each had a time of 53 milliseconds. There are way too many factors beyond your provider's control to make that a worthwhile upgrade. –  Mike Renfro Jun 3 '11 at 12:33
1  
+1 Technically there is a difference, however it is completely inappreciable unless the particular application is incredibly sensitive to latency. –  Chris S Jun 3 '11 at 12:34
    
Thank you for the test! From 0.21 to 0.17 is an improvement of 20%, which is great. I'm not concerned of the ping from home (50ms) but the time the request stays at the provider. We tweaked all processing (CPU) and non drive-IO (RAM/Cache/etc.) to the max, so now I question how much the network speed between the servers adds to the whole latency as total. E.g. we make ~20 Redis-Requests for one webserver-request. @MikeRenfro: can you do the same test with a higher loadsize? Normal ping is 30byte, avg. Redis around 250. I would expect the difference to grow. –  Andreas Richter Jun 3 '11 at 13:21
    
@Andreas - yes, technically it was a 20% improvement. However, when you look at the entire system and all the variables that come into play when calculating latency (application logic, IO latency, etc.), that 20% becomes a tiny fraction of 1%. –  EEAA Jun 3 '11 at 14:05
2  
@Andreas; I think you totally missed the point of those comments. That's an improvement of 40 nanoseconds. An amount that is completely imperceivable to human beings. And that's not a cumulative number, it's not like each request takes 40 nanoseconds longer; it's just the first will be that much quicker, so the second will be lined up right behind it either way. –  Chris S Jun 3 '11 at 14:05

I think you have a fundamental misconception about bandwidth latency and "speed". Speed is a function of bandwidth and latency. For instance consider a shipment of data on DVDs shipped across the country taking 3 days to arrive. Compare that to sending the data across the internet. The internet has a much lower latency connection, but to match the "speed" of the connection to the shipment you'd have to have recieve at 9.6MB a sec (reference example from this source).

In your case upgrading to higher bandwidth would allow you to serve more concurrent users but not improve the latency to any individual user.

share|improve this answer
    
That is incorrect - simply compare the ping with differnt payload that is below the current MTU: ping server -i0.05 -c200 -s30 vs. ping server -i0.05 -c200 -s300 ... Or speaking in your example: the truck with 1mio DVDs would drive slower, because it is heavier than the one with 1 DVD. –  Andreas Richter Jun 3 '11 at 13:25
2  
@andreas ping time isn't the whole story - so lets take fo rthe sake of argument that packets lower than the MTU arrive faster than packets at full MTU. the differenc eis you don't have all the data that the 1 packet has at full mtu in the same amount of time that the 2 packets take to arrive. The latency is the time taken for all the data to arrive. to go back to the truck analogy, even if it takes a truck with 1 Cd to arrive in half the time as the truck carrying 500 cds it still takes that truck 500 trips to deliver the data (750 days vs 3). –  Jim B Jun 3 '11 at 13:44
    
@JimB: yes, as mentioned my question was not about loadsize, but about the speed: the full truck needs 10 hour to be scanned by customs, the small one only 1 hour. 100mbit/s also means, that a 100bit packet needs a theoretical minimum of 0,000954ms and a 1000bit packet a theoretical minimum of 0,00954ms. Of course processing time/etc. on the network card/switch/etc. make a higher chunk of the whole latency, but I would also expect these to be faster in a 1gbit switch, etc. Please see the comment by @MikeRenfro, he actually tested it and came to a 20% increase. –  Andreas Richter Jun 3 '11 at 13:52
    
@andreas - 20% on the same subnet which is irrlevant to your question –  Jim B Jun 3 '11 at 14:19
1  
@andreas 20% of a sub-millisecond ping is still a sub-millisecond ping. Even 150% of a sub-millisecond ping, as in your testing, is not going to matter in the real world. Do you really have an application where it matters whether your data got there in 0.0003 seconds instead of 0.0002 seconds? –  Shane Madden Jun 3 '11 at 14:40

Let's assume 300 byte packets (if you use -s 300 it would be actually bigger because of headers).

300 bytes = 2400 bits
2400 bits / 100Mbit/sec = 0.024ms

0.024ms is approximately the actual time required to send the frame (not counting medium access time nor headers).

In a ping-pong sequence it would take twice that time (0.048ms), plus the time for the OS to process the query.

That means to me that your latency is caused by a 90% of several overhead factors. I'm not sure if you will see much difference with Gb. A probably less than 1ms difference will not on be noticeable a webserver.

Anyway could you try with a really big packet like 1400 byte?

share|improve this answer
    
Someone already ran the numbers for a particular server and the difference came back at 40 nanoseconds. Your guesstimate is off by a factor of 25 times too large. –  Chris S Jun 3 '11 at 14:08
    
@LatinSuD: thank you for the constructive approach and not blaming that I don't know what I'm doing. I will post the results in the actual question since I can do formating there. But btw. I would also expect the 90% overhead to have a speed increase, since the processors in the network cards, etc. are faster for GBit aswell (hopefully). @ChrisS: micro-seconds and I don't understand what you mean with the 25. –  Andreas Richter Jun 3 '11 at 15:13
    
My apologies for mixing-up micro and nano; in any case it's imperceivable. LatinSuD guesstimated a difference of 1 whole millisecond, which it 25 times more than the 40 microseconds found by Mike. –  Chris S Jun 3 '11 at 15:35
    
@ChrisS: no worries. The 0,04ms was for a 38 byte ping, if our average server-server packet is around 300 byte, the difference could be 0,4ms. If we now make 20 requests for one webserver-requst (Redis, MySQL, etc.), this would lead to a 8ms speed increase which would be a 10% speed increase for current web-requests and would totally justify the additional costs. I simply dont have the ressources here to run the tests myself, but belive me, that even if it is not perceivable by humans, it still can be of relevance. Like electricity or god. –  Andreas Richter Jun 3 '11 at 15:42
1  
@Andreas, I highly doubt it will scale like that; both a 10x larger packet being 10x less latency and 20x as many packets being 20x faster. If it does pan out, that's a 10% reduction in network overhead, you still have to account for the time spent in the application, which is likely to be one to several orders of magnitude longer than the network latency component. I hope it works out well for you in any case. –  Chris S Jun 3 '11 at 17:04

This depends on the type of switch you're connecting to. On some vendors (such as Crisco... I mean Cisco), ICMP packets will flow back to the CPU (gag).

You may find a better test would be to perform a host-to-host test using a 100Mbps and 1Gbps link (i.e. not a host-to-switch or host-to-router test).

At the end of the day, the latency is going to come down to the forwarding rate on the switch and the particulars of the switch's architecture (where the ASICs are placed on the board, how locking is handled between line cards, etc). Good luck with your testing.

share|improve this answer
    
Thank you, I only refer to Host-Switch-Host tests and I don't understand all the switch interna. I would simply love to see, if somebody ever benchmarked: Host-(100mbit)-Switch-(100mbit)-Host, Host-(100mbit)-Switch-(1gbit)-Host and Host-(1gbit)-Switch-(1gbit)-Host latency for different packet sizes. If nobody did, I will do it and post the answer here. –  Andreas Richter Jun 3 '11 at 15:35
1  
I used to resell switch gear. Suffice it to say, your findings suggest to me that you're plugged in to a Cisco switch. There are other alternatives that provide lower latencies. As you rightly pointed out, more bandwidth doesn't translate in to lower latencies (Comcast is the primary culprit for making people dumb in this regard). Given you're in a hosted environment, you're probably stuck with their hardware (and since in a hosted environment, the extra few microsec aren't terribly crucial). Show me millions of pps at steady-state and I'll get interested in providing more details. :) –  Sean Jun 3 '11 at 15:44

You're looking at the world through a pinhole. A valid test of latency differences at different speeds would be between two identical NICs connected with a cross-connect cable. Set the NICs mathching speeds of 10mb, 100mb and 1000mb. This will show that there is virtually no difference in latency at the different speeds. All packets travel at the same wire speed regardless of max bandwidth being used. Once you add switches with store and forward caching everything changes. Testing latency through a switch must be done with only two connections to the switch. Any other traffic may affect the latency of your test. Even then the switch may roll-over logs, adjust packet type counters, update internal clock, etc.. Everything may affect latency.

Yes, switching from 100mb to 1gb might be faster (lower latency) due to hardware changes, different NIC, different switch, different driver. I have seen larger changes in ping latency from driver differences than any other changes; bandwidth, switches, offloading NICs, etc..

The switch would be the next biggest change with cut-through significantly faster than store and forward for single transmit tests. However, a well designed store and forward switch may overtake the cut-through switch in overall performance under high load. In the early days of gigabit I'v seen 10mb high performance backplane switches with lower latency than cheap gigabit switches.

Ping tests are practically irrelevant for performance analysis when using the Internet. They are quick tests to get a ballpark idea of what's happening on the transport at the moment of the test. Production performance testing is much more complicated than just a ping. High performance switches are computers and under high load behave differently - change in latency.

Having a slower NIC, or a NIC set to a slower speed, could actually help a server with concurrent bursts by throttling the input to the server using the switches cache. A single re-transmit may negate any decrease in latency. Usually medium to high-load traffic levels are important, not single ping tests. e.g. Old slow Sun Ultrasparc (higher latency for a single ping) outperforms new cheap gigabit desktop used as dev server when under 70% 100mb bandwidth load. Desktop has faster gb NIC, faster connection gb-gb, faster memory, more memory, faster disk and faster processor but it doesn't perform as well as tuned server class hardware/software. This is not to say that a current tuned server running gb-gb isn't faster than old hardware, even able to handle larger throughput loads. There is just more complexity to the question of "higher performance" than you seem to be asking.

Find out if your provider is using different switches for the 100mb vs. 1gb connections. If they use the same switch backplane than I would only pay for the increase if the traffic levels exceeded the lower bandwidth. Otherwise you may find that in short time many other users will switch over to the gigabit and the few users left on the old switch now have higher performance - lower latency, during high loads on the switch (overall switch load, not just to your servers).

Apples and oranges example: Local ISP provided a new switch for bundled services, DSL and phone. Initially users saw an increase in performance. System was oversold. Now users that remain on the old switch have higher consistent performance. During late night, users on the new system are faster. In the evening under high load the old switch clients clearly outperform the new overloaded system.

Lower latency doesn't always correlate to faster delivery. You mention MySQl in the 20 requests to serve a single page. That traffic shouldn't be on the same NIC as the page requests. Moving all internal traffic to an internal network will reduce collisions and total packet counts on the outgoing NIC and provide larger gains than the .04ms latency gain of a single packet. Reduce the number of requests per page to reduce page load latency. Compress the pages, html, css, javascript, images to decrease page load times. These three changes will give larger overall gains ongoing than paying for bandwidth not being used to get a .04ms latency reduction. The ping needs to run 24hrs and be averaged to see the real latency change. Smart switches now do adaptive RTSP type throttling with small initial bandwidth increases and large transfers throttled. Depending on your page sizes (graphics, large html/css/javascript) you may see initial connection latencies/bandwidth much lower/higher than a large page or full page transfers. If part of your page is streaming you may see drastically different performance between the page and the stream.

share|improve this answer
    
Thank you for all the great input: 1.) It's the same switch, 2.) A second NIC for internal/external sounds resonable and worth a try - even though e.g. MySQL/etc. are all bi-directional, so collisions would "only" be reduced to half, 3.) A realworld test is preferable to only Nic-Nic, Mike has done it with a subnet, and got what you expected reg. hardware: "56 byte = 19% improvement, 200 byte = 27%, 1000 byte = 59%! So the larger the packet, the more it matters. And Gigabit increased only from 0.17ms (56 byte) to 0.23ms (1000 byte) => 35%, while 100 Mbit increased from 0.21 to 0.56 => 166%". –  Andreas Richter Jun 6 '11 at 12:01
up vote 4 down vote accepted

YES gbit has a lower latency, since:

  • the same number of bytes can be transfered in faster time

BUT the improvement is only appreciable if the packet(s) have a certain size:

  • 56 byte package => virtually no faster transfer
  • 1000 byte package => 20% faster transfer
  • 20000 byte package(s) => 80% faster transfer

So if you have an application which is very sensitive to latency (4ms vs. 0.8ms, round-trip for 20kb) and require larger packages to be transferred, than switching from 100mbit to gbit can give you a latency reduction, even though you use much less than the 100mbit/s in average (= the link is not saturated permanently).

Server (100mbit) -> Switch (gbit) -> Server (100mbit):

size: 56 :: rtt min/avg/max/mdev = 0.124/0.176/0.627/0.052 ms
size: 100 :: rtt min/avg/max/mdev = 0.131/0.380/1.165/0.073 ms
size: 300 :: rtt min/avg/max/mdev = 0.311/0.463/2.387/0.115 ms
size: 800 :: rtt min/avg/max/mdev = 0.511/0.665/1.012/0.055 ms
size: 1000 :: rtt min/avg/max/mdev = 0.560/0.747/1.393/0.058 ms
size: 1200 :: rtt min/avg/max/mdev = 0.640/0.830/2.478/0.104 ms
size: 1492 :: rtt min/avg/max/mdev = 0.717/0.782/1.514/0.055 ms
size: 1800 :: rtt min/avg/max/mdev = 0.831/0.953/1.363/0.055 ms
size: 5000 :: rtt min/avg/max/mdev = 1.352/1.458/2.269/0.073 ms
size: 20000 :: rtt min/avg/max/mdev = 3.856/3.974/5.058/0.123 ms

Server (gbit) -> Switch (gbit) -> Server (gbit):

size: 56 :: rtt min/avg/max/mdev = 0.073/0.144/0.267/0.038 ms
size: 100 :: rtt min/avg/max/mdev = 0.129/0.501/0.630/0.074 ms
size: 300 :: rtt min/avg/max/mdev = 0.185/0.514/0.650/0.072 ms
size: 800 :: rtt min/avg/max/mdev = 0.201/0.583/0.792/0.079 ms
size: 1000 :: rtt min/avg/max/mdev = 0.204/0.609/0.748/0.078 ms
size: 1200 :: rtt min/avg/max/mdev = 0.220/0.621/0.746/0.080 ms
size: 1492 :: rtt min/avg/max/mdev = 0.256/0.343/0.487/0.043 ms
size: 1800 :: rtt min/avg/max/mdev = 0.311/0.672/0.815/0.079 ms
size: 5000 :: rtt min/avg/max/mdev = 0.347/0.556/0.803/0.048 ms
size: 20000 :: rtt min/avg/max/mdev = 0.620/0.813/1.222/0.122 ms

= in average over multiple servers 80% latency reduction for 20kb ping

(If only one of the links is gbit, you will still have a 5% latency reduction for 20kb ping.)

share|improve this answer
    
With most networking gear being store and forward a packet has to be fully received by a switch/router before it's passed on. Faster connections reduce this time which also reduces latency (as long as the connection doesn't get the speed from several parallel links). –  Brian Nov 19 '12 at 18:23
    
Due to the explanation, this answer is by far the best on the page. The others seem to want to explain away this fact by assuming a special case - network performance across a long distance/lots of switches. That is important to consider, especially considering the OP's concern (webserver), but this answer also shows how much of a difference switch speed can make in a single hop. –  Tyler Sep 5 '13 at 18:49

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.