We have a legacy web application that is running in our data center on premises located in Houston. We have a developed a new .net 4 based web application in order to provide new features to customers. The new web application is hosted in amazon aws ec2 environment (N. Virginia region us-east-1b zone).

In order to get seamlessly integrate with the legacy application the new web application makes web service calls to retrieve data. We are seeing an unusually high latency time in the order of 5+ seconds for these web service calls. The exact same web service call returns in less than a second on our local PCs (which makes sense given physical proximity to the actual server). The weird part is that we have developers in California who also have the same milliseconds response time. We are testing the web service response using third party tools such as SoapUI, Google Chrome extensions such as Advanced REST Client, Postman REST Client, etc.

As if this wasnt weird enough, we have noticed the same low latency from certain other ec2 instances while testing which are in the same region and availability zone as well. If we experienced the high latency consistently from all the ec2 instances I could understand. But there is something else going on.

Comparing the various stats and results between the low latency and high latency ec2 servers do not show any significant differences: ping (constant 40ms), tracert, winmtr, etc.

We have instances that are in the VPC as well. So I tried both the public and private IP address of the web service host server and that didnt make a difference either for the above results.

We need to resolve this latency issue as this is causing the resulting web pages to load very slowly (almost 15+ seconds which is simply unacceptable).

The ec2 instances have Windows Server Datacenter 64 bit. Let me know if there is any other infor I can provide to help diagnose this.

  • Interesting, I noticed that when I restarted the ec2 instance that was showing low latency initially, it is now showing high latency. – SibzTer Nov 12 '12 at 17:09

First, this sounds like something you will have to troubleshoot with Amazon. A simple tracert to your destination host will show you where the delay is.

Second, you should design your application to load the page and then display the data whenever it becomes available, not simply expect it to show up. It sounds to me like you have not utilized any AJAX failover in your application. How can you be sure that your new application will ALWAYS have a connection back to your legacy app.

TL;DR: Run a tracert, call Amazon, check your code, make sure it allows for delays and no internet connection scenarios.

  • We are working with Amazon EC2 engineers for almost the past 2 weeks now. We have given them ping, tracert, winmtr, etc. outputs from both ends. These results included outputs from ec2 instances with low and high latency. But they havent been able to figure out the issue yet. – SibzTer Nov 11 '12 at 23:06
  • One thing I have noticed is that the ec2 instance with low latency has .net v3.5 installed in it whereas the high latency instance has .net v4.5. That was the only difference configuration wise between the two instances. Further, when I installed .net v4.5 on the low latency instance, it started experiencing high latency as well. And also to rule out the issue with our application code, we are testing with third party REST clients such as SOAP UI, chrome extensions, etc. – SibzTer Nov 11 '12 at 23:11

The 5+ second latency looks like a rare but not unusual event. Try using something like iperf to check the bandwidth between your datacenter at Houston and the ec2 instance at Virginia. If it helps, here are some of the bandwidth measurements we made recently, between two amazon regions (east and west), and between these regions and our datacenter host in California:

ec2-east -- ec2-west: 25.75 +-7.96
ec2-east -- localhost: 665.0 +-104.5
ec2-west -- localhost: 79.35 +-4.38

All numbers are in Mbits/sec, measured over twenty times and averaged. +- is the stdev.

Hope this helps.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.