up vote 39 down vote favorite
30
share [g+] share [fb]

At the moment we're trying to decide whether to move our datacenter from the west coast to the east coast.

However, I am seeing some disturbing latency numbers from my west coast location to the east coast. Here's a sample result, retrieving a small .png logo file in Google Chrome and using the dev tools to see how long the request takes:

  • West coast to east coast:
    215 ms latency, 46 ms transfer time, 261 ms total
  • West coast to west coast:
    114 ms latency, 41 ms transfer time, 155 ms total

some URLs if you want to try yourself:

It makes sense that Corvallis, OR is geographically closer to my location in Berkeley, CA so I expect the connection to be a bit faster.. but I'm seeing an increase in latency of +100ms when I perform the same test to the NYC server. That seems .. excessive to me. Particularly since the time spent transferring the actual data only increased 10%, yet the latency increased 100%!

That feels... wrong... to me.

I found a few links here that were helpful (through Google no less!) ...

... but nothing authoritative.

So, is this normal? It doesn't feel normal. What is the "typical" latency I should expect when moving network packets from the east coast <--> west coast of the USA?

link|improve this question
84% accept rate
1  
Any measurement across Networks you do not control seem almost pointless. Too often in these types of Network discussions it seems that we forget there is a temporal component associated with every packet. If you ran the test repeatedly 24 x 7 and arrived at some conclusion that is one thing. If you ran the test twice then I suggest you run it some more. And to those advocating the use of ping as some measure of performance, don't. On every major network I ever worked on we set ICMP traffic to the lowest priority. Ping means only one thing, and it ain't ;) about performance. – dbasnett May 9 '10 at 11:03
From where I live, Jefferson City, MO, the times are similar. – dbasnett May 9 '10 at 12:46
feedback

8 Answers

up vote 56 down vote accepted

Speed of Light:
You are not going beat the speed of light as an interesting academic point. This link works out Stanford to Boston at ~40ms best possible time. When this person did the calculation he decided the internet operates at about "within a factor of two of the speed of light", so there is about ~85ms transfer time.

TCP Window Size:
If you are having transfer speed issues you may need to increase the receiving window tcp size. You might also need to enable window scaling if this is a high bandwidth connection with high latency (Called a "Long Fat Pipe"). So if you are transferring a large file, you need to have a big enough receiving window to fill the pipe without having to wait for window updates. I went into some detail on how to calculate that in my answer Tuning an Elephant.

Geography and Latency:
A failing point of some CDNs (Content Distribtuion Networks) is that they equate latency and geography. Google did a lot of research with there network and found flaws in this, they published the results in the white paper Moving Beyond End-to-End Path Information to Optimize CDN Performance:

First, even though most clients are served by a geographically nearby CDN node, a sizeable fraction of clients experience latencies several tens of milliseconds higher than other clients in the same region. Second, we find that queueing delays often override the benefits of a client interacting with a nearby server.

BGP Peerings:
Also if you start to study BGP (core internet routing protocol) and how ISPs choose peerings, you will find it is often more about finances and politics, so you might not always get the 'best' route to certain geographic locations depending on your ISP. You can look at how your IP is connected to other ISPs (Autonomous Systems) using a looking glass router. You can also use a special whois service:

whois -h v4-peer.whois.cymru.com "69.59.196.212"
PEER_AS | IP               | AS Name
25899   | 69.59.196.212    | LSNET - LS Networks
32869   | 69.59.196.212    | SILVERSTAR-NET - Silver Star Telecom, LLC

It also fun to explore these as peerings with a gui tool like linkrank, it gives you a picture of the internet around you.

link|improve this answer
agree, speed of light as the crow flies is the best you can possibly do. – Jeff Atwood♦ Apr 30 '10 at 12:07
really excellent answer by the way, this is exactly what I was looking for. Thank you. – Jeff Atwood♦ Apr 30 '10 at 12:29
2  
For the curious the actual math is: 3000 mi / c = 16.1ms – tylerl May 4 '10 at 5:20
7  
In a vacuum a photon can travel the equator in roughly 134 ms. The same photon in glass would take around 200 ms. A 3,000 mile piece of fiber has 24 ms. of delay without any devices. – dbasnett May 9 '10 at 12:33
2  
Google did a lot of research with their network – Rich Seller May 11 '10 at 4:49
feedback
up vote 13 down vote

This site would suggest around 70-80ms latency between East/West coast US is typical (San Francisco to New York for example).

Trans-Atlantic Path
NY      78    London
Wash    87    Frankfurt

Trans-Pacific Path
SF     147    Hong Kong

Trans-USA Path
SF      72    NY

Here are my timings (I'm in London, England, so my West coast times are higher than East). I get a 74ms latency difference, which seems to support the value from that site.

NY - 108ms latency, 61ms transfer, 169 total
OR - 182ms latency, 71ms transfer, 253 total

These were measured using Google Chrome dev tools.

link|improve this answer
cool chart! NY to SF is currently 71 ms on it, so you're right -- we can't expect to do better than that. – Jeff Atwood♦ Apr 30 '10 at 11:52
feedback
up vote 7 down vote

Measure with ICMP first if at all possible. ICMP tests typically use a very small payload by default, do not use a three-way handshake, and do not have to interact with another application up the stack like HTTP does. Whatever the case, it is of the utmost importance that HTTP results do not get mixed up with ICMP results. They are apples and oranges.

Going by the answer of Rich Adams and using the site that he recommended, you can see that on AT&T's backbone, it takes 72 ms for ICMP traffic to move between their SF and NY endpoints. That is a fair number to go by, but you must keep in mind that this is on a network that is completely controlled by AT&T. It does not take into account the transition to your home or office network.

If you do a ping against careers.stackoverflow.com from your source network, you should see something not too far off of 72 ms (maybe +/- 20 ms). If that is the case, then you can probably assume that the network path between the two of you is okay and running within normal ranges. If not, don't panic and measure from a few other places. It could be your ISP.

Assuming that passed, your next step is to tackle the application layer and determine if there is anything wrong with the additional overhead you are seeing with your HTTP requests. This can vary from app to app due to hardware, OS, and application stack, but since you have roughly identical equipment on both the East and West coasts, you could have East coast users hit the West coast servers and West coast users hit the East coast. If both sites are configured properly, I would expect to see all numbers to be more less equal and to therefore demonstrate that what you are seeing is pretty much par for the coarse.

If those HTTP times have a wide variance, I would not be surprised if there was a configuration issue on the slower performing site.

Now, once you are at this point, you can attempt to do some more aggressive optimization on the app side in order to see if those numbers can be reduced at all. For example, if your are using IIS 7, are you taking advantage of its caching capabilities, etc? Maybe you could win something there, maybe not. When it comes to tweaking low-level items such as TCP windows, I am very skeptical that it would have much of an impact for something like Stack Overflow. But hey - you won't know until you try it and measure.

link|improve this answer
feedback
up vote 4 down vote

I'm seeing consistent differences, and I'm sitting in Norway:

serverfault       careers
  509ms            282ms
  511ms            304ms
  488ms            295ms
  480ms            274ms
  498ms            278ms

This was measured with the scientific accurate and proven method of using the resources view of Google Chrome and just repeatedly refreshing each link.

Traceroute to serverfault

Tracing route to serverfault.com [69.59.196.212]
over a maximum of 30 hops:

  1    <1 ms     1 ms    <1 ms  81.27.47.1
  2     2 ms     1 ms     1 ms  qos-1.webhuset.no [81.27.32.17]
  3     1 ms     1 ms     1 ms  81.27.32.10
  4     1 ms     2 ms     1 ms  201.82-134-26.bkkb.no [82.134.26.201]
  5    14 ms    14 ms    14 ms  193.28.236.253
  6    13 ms    13 ms    14 ms  TenGigabitEthernet8-4.ar1.OSL2.gblx.net [64.209.94.125]
  7    22 ms    21 ms    21 ms  te7-1-10G.ar3.cph1.gblx.net [67.16.161.93]
  8    21 ms    20 ms    20 ms  sprint-1.ar3.CPH1.gblx.net [64.212.107.18]
  9    21 ms    21 ms    20 ms  sl-bb20-cop-15-0-0.sprintlink.net [80.77.64.33]
 10   107 ms   107 ms   107 ms  144.232.24.12
 11   107 ms   106 ms   105 ms  sl-bb20-msq-15-0-0.sprintlink.net [144.232.9.109]
 12   106 ms   106 ms   107 ms  sl-crs2-nyc-0-2-5-0.sprintlink.net [144.232.20.75]
 13   129 ms   135 ms   134 ms  sl-crs2-chi-0-15-0-0.sprintlink.net [144.232.24.208]
 14   183 ms   183 ms   184 ms  sl-crs2-chi-0-10-3-0.sprintlink.net [144.232.20.85]
 15   189 ms   189 ms   189 ms  sl-gw12-sea-2-0-0.sprintlink.net [144.232.6.120]
 16   193 ms   189 ms   189 ms  204.181.35.194
 17   181 ms   181 ms   180 ms  core2-gi61-to-core1-gi63.silverstartelecom.com [74.85.240.14]
 18   182 ms   182 ms   182 ms  sst-6509b-gi51-2-gsr2-gi63.silverstartelecom.com [74.85.242.6]
 19   195 ms   195 ms   194 ms  sst-6509-peak-p2p-gi13.silverstartelecom.com [12.111.189.106]
 20   197 ms   197 ms   197 ms  ge-0-0-2-cvo-br1.peak.org [69.59.218.2]
 21   188 ms   187 ms   189 ms  ge-1-0-0-cvo-core2.peak.org [69.59.218.193]
 22   198 ms   198 ms   198 ms  vlan5-cvo-colo2.peak.org [69.59.218.226]
 23   198 ms   197 ms   197 ms  stackoverflow.com [69.59.196.212]

Trace complete.

Traceroute to careers

Tracing route to careers.stackoverflow.com [64.34.80.176]
over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms  81.27.47.1
  2     2 ms     1 ms    <1 ms  qos-1.webhuset.no [81.27.32.17]
  3     1 ms     1 ms     1 ms  81.27.32.10
  4     1 ms     1 ms     2 ms  201.82-134-26.bkkb.no [82.134.26.201]
  5    12 ms    13 ms    13 ms  193.28.236.253
  6    13 ms    14 ms    14 ms  TenGigabitEthernet8-4.ar1.OSL2.gblx.net [64.209.94.125]
  7    21 ms    21 ms    21 ms  ge7-1-10G.ar1.ARN3.gblx.net [67.17.109.89]
  8    21 ms    20 ms    20 ms  tiscali-1.ar1.ARN3.gblx.net [64.208.110.130]
  9   116 ms   117 ms   122 ms  xe-4-2-0.nyc20.ip4.tinet.net [89.149.184.142]
 10   121 ms   122 ms   121 ms  peer1-gw.ip4.tinet.net [77.67.70.194]
 11     *        *        *     Request timed out.

Unfortunately, it now starts going into a loop or whatnot and continues giving stars and timeout until 30 hops and then finishes.

Note, the traceroutes are from a different host than the timings at the start, I had to RDP to my hosted server to execute them

link|improve this answer
1  
right, it is expected that the east coast datacenter would be more friendly to our European audiences -- you're seeing about +200ms time taken to traverse the width of the USA. Should only be ~80ms per the other answers though? – Jeff Atwood♦ Apr 30 '10 at 11:49
it looks like it is consistent at around 200ms, I've hit refresh about 20-30 times now on both (not at the same time though), and the serverfault site looks like it hovers around 200ms +/- more than the other one. I tried a traceroute, but it comes up with stars on everything so perhaps our IT admins have blocked something. – Lasse V. Karlsen Apr 30 '10 at 11:59
feedback
up vote 2 down vote

I see approx 80-90ms latency on well run, well measured links between East and West coasts.

It would be interesting to see where you're gaining latency - try a tool like layer-four traceroute (lft). Chances are a lot of it is gained on the "last mile" (i.e. in your local broadband provider).

That the transfer time was only slightly impacted is to be expected - packet loss and jitter are more useful measurements to look at when investigating transfer time differences between two locations.

link|improve this answer
feedback
up vote 2 down vote

Just for fun, when I played the online game Lineage 2 NA release from within Europe:

Response time to east coast servers: ~110-120ms
Response time to west coast servers: ~190-220ms

The difference seems to support that up to 100ms is within reason, considering the unpredictable nature of the internet.

Using the acclaimed Chrome refresh test, I get document load time that differs with roughly 130ms.

link|improve this answer
feedback
up vote 1 down vote

NYC Timings:

NY     OR
109ms  271ms
72ms   227ms
30ms   225ms
33ms   114ms
34ms   224ms

Using Chrome, on a residential connection.

Using lft from a VPS in a datacenter in Newark, New Jersey:

terracidal ~ # lft careers.stackoverflow.com -V
Layer Four Traceroute (LFT) version 3.0
Using device eth0, members.linode.com (97.107.139.108):53
TTL LFT trace to 64.34.80.176:80/tcp
 1  207.192.75.2 0.4/0.5ms
 2  vlan804.tbr2.mmu.nac.net (209.123.10.13) 0.4/0.3ms
 3  0.e1-1.tbr2.tl9.nac.net (209.123.10.78) 1.3/1.5ms
 4  nyiix.Peer1.net (198.32.160.65) 1.4/1.4ms
 5  oc48-po3-0.nyc-75bre-dis-1.peer1.net (216.187.115.134) 1.6/1.5ms
 6  216.187.115.145 2.7/2.2ms
 7  64.34.60.28 2.3/1.8ms
 8  [target open] 64.34.80.176:80 2.5ms

terracidal ~ # lft serverfault.com -V
Layer Four Traceroute (LFT) version 3.0
Using device eth0, members.linode.com (97.107.139.108):53
TTL LFT trace to stackoverflow.com (69.59.196.212):80/tcp
 1  207.192.75.2 36.4/0.6ms
 2  vlan803.tbr1.mmu.nac.net (209.123.10.29) 0.4/0.4ms
 3  0.e1-1.tbr1.tl9.nac.net (209.123.10.102) 1.3/1.4ms
 4  nyk-b3-link.telia.net (213.248.99.89) 1.6/1.4ms
 5  nyk-bb2-link.telia.net (80.91.250.94) 1.9/84.8ms
 6  nyk-b5-link.telia.net (80.91.253.106) 1.7/1.7ms
 7  192.205.34.53 2.1/2.1ms
 8  cr1.n54ny.ip.att.net (12.122.81.106) 83.5/83.6ms
 9  cr2.cgcil.ip.att.net (12.122.1.2) 82.7/83.1ms
10  cr2.st6wa.ip.att.net (12.122.31.130) 83.4/83.5ms
11  cr2.ptdor.ip.att.net (12.122.30.149) 82.7/82.7ms
12  gar1.ptdor.ip.att.net (12.123.157.65) 82.2/82.3ms
13  12.118.177.74 82.9/82.8ms
14  sst-6509b-gi51-2-gsr2-gi63.silverstartelecom.com (74.85.242.6) 84.1/84.0ms
15  sst-6509-peak-p2p-gi13.silverstartelecom.com (12.111.189.106) 83.3/83.4ms
16  ge-0-0-2-cvo-br1.peak.org (69.59.218.2) 86.3/86.2ms
**  [neglected] no reply packets received from TTLs 17 through 18
19  [target closed] stackoverflow.com (69.59.196.212):80 86.3/86.3ms
link|improve this answer
feedback
up vote 0 down vote

I get about 200-250ms to WoW servers in NY. (I’m in Spain).

I don’t think we’re near a solution that will accelerate this. (sadly)

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.