Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have two sites hosted on two different data-centres. Recently one site has become very slow. The ping response from application server to DB server is not fast enough. How do I investigate the issue?

On fast server:
10 packets transmitted, 10 received, 0% packet loss, time 8998ms
rtt min/avg/max/mdev = 0.243/0.279/0.502/0.074 ms

On slow server:
21 packets transmitted, 21 received, 0% packet loss, time 20011ms
rtt min/avg/max/mdev = 1.131/1.816/3.584/0.560 ms

The tracert command shows the following:

On fast server:
tracert db
traceroute to db (xxx.xxx.100.101), 30 hops max, 40 byte packets
 1  db (xxx.xxx.100.101)  0.552 ms  0.530 ms  0.527 ms

 On slow server:
tracert xxx.16.55.140
traceroute to xxx.16.55.140 (xxx.16.55.140), 30 hops max, 40 byte packets
 1  xxx.16.55.140 (xxx.16.55.140)  1.859 ms  1.845 ms  1.842 ms
share|improve this question
1  
1.8 ms is not fast enough? All but the most extremely performance-sensitive applications could easily tolerate that latency. How far away is the location? –  Shane Madden Aug 18 '11 at 4:49
    
Scream if you want to go FASTER!! –  Wesley Aug 18 '11 at 5:04
    
What are the load averages on the slow servers? –  Matt Aug 18 '11 at 5:06

6 Answers 6

up vote 2 down vote accepted

Perform a pathping from the web server to the database server and see where the slowdown is reported to be. Then, confirm by performing a pathping from the database server to the web front end. Use the IP address of the nodes, and not DNS names. As Womble pointed out, it could be rDNS slowdowns.

FYI, pathping, like tracert, can provide deceiving path information simply based on how packets might be routed one way forward and a different way backward based on network congestion. Also, the forward path is not guaranteed to be the same with each increased hop. However, those are extraneous topics at this point. Moving on...

Once you determine where the slowdown is you can continue to troubleshoot. It may be the end nodes themselves are the slowdown if they are under heavy load or improperly configured in some way. If you find out what the slow node is, update your questions with the proper information.

share|improve this answer

You can use traceroute to see if there is a point along the path that is slowing everything down.

share|improve this answer

Traceroute (mtr is even better) the path between the two machines, looking for particular hops that add a lot of latency. Once you've identified the location, you can look to the cause (check port stats on both sides of the link in question to see if there's queueing or some other problem); you're not dropping packets (well, not excessive numbers of them -- 21 pings isn't exactly statistically significant) so you're probably not overflowing buffers anywhere.

However, you're still only seeing 1.8ms of latency for the "slower" link, which really is excellent over any sort of WAN link. Unless you're doing something incredibly latency-sensitive (like high-speed trading), I'm struggling to imagine how that could be "very slow" in any meaningful sense.

share|improve this answer

10 packets transmitted, 10 received, 0% packet loss, time 8998ms

8998ms is huge network latency. You can use mtr to see if it is failing on some point? How far away is the data center location? Is it connecting to China from USA? What is the average load of the server?

share|improve this answer
1  
8998ms would be a huge network latency, but that's not what that figure is in the ping output above. 8998ms is the total time taken to transmit 10 pings with a 1 second gap in between them. You might also notice that it took about 20 seconds to transmit 21 pings to the slow server. –  Ladadadada Aug 18 '11 at 9:03

You state in your question that the site has become slow and then ask about ping times. Is it possible that the site is slow for other reasons ?

If you are hosting two websites in two different datacentres with only one database, the bandwidth between the two datacentres could be the limiting factor.

It might be worth checking how much data you are pulling over from the database in each query. It's not unusual to see 10MB come back in a database query only to have the scripting language parse/mangle/throw away the data until there's only a few KB left to send to the user. Plenty of people use "SELECT *" even when they only need one field. It's also worth checking how much traffic you can see on your database port overall. If you only have a 10Mb link to the other datacentre and you are pulling back even a 1MB query, it will take almost a second to arrive.

If latency is actually your issue rather than bandwidth, using persistent connections can help because it avoids making a whole new tcp connection for each query. Setting up a slave read-only database in the second datacentre can also help as read-only queries can be done locally.

share|improve this answer

Standard deviation (mdev) of packets on "slow" is high relative to avg. I'd say the network is congested (either on the host level or the switch/router)

You can try using iperf in UDP mode, you'll get amount of jitter this way.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.