Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a VPS Server running WHM 11.36.0 (build 14) and CENTOS 6.4 x86_64 xenpv – control

I have a number of accounts on my server and whenever I make a request to one of them, the request takes a good few seconds to establish. When transfers have connected, it's fast, but it's that initial 20 seconds of 'nothing happening' that's worrying me.

We make heavy use of our VPS in my organisation and I can't really afford for slowdown.

Does anyone have any idea why this is happening?
I'm not very technically experienced with servers (I'm a software guy) so I'm not sure how to configure servers correctly, to me, it looks like a DNS error, but I have no idea how to debug or repair these issues! Can anyone help?

Here are two websites of which currently experience this 'hang' -

UPDATE:

I've spoken to my VPS hosts, and the official line is that the hang is caused by my local DNS rather than the DNS of the server - but obviously, this issue is only occurring when trying to access an account on our own server and nowhere else on the web.

To test this, I changed my local DNS to 8.8.8.8 and 8.8.4.4 and then repeated this change on our router. There doesn't seem to be any change in behaviour at all.

I'm desperate to get this resolved but as yet, to no avail!

share|improve this question
    
In trying to isolate the network factor as whole to see if it's really related or not, can you check if the issue happens when connecting to the same websites from the localhost? You can use 'curl' if you're going to do it from the command line, ex: curl localhost or curl sparebrainmanagementsupport.com –  minniux Mar 20 '13 at 18:08

3 Answers 3

To check if it is a DNS or a TCP-connection issue I'd install tcpdump on the server, Wireshark on your client and run some measurements yourself. This needs some console skills though, but it's worth a try.

We will capture raw traffic from your network interface and look into the bowels of the wire to figure out, if it's a DNS or a TCP-problem. However, this won't solve the root cause of your problem, it will only give you pointers were to look further.

This whole posting is based on the assumption, that it's either a DNS reverse-lookup timeout problem or a deeper problem on the network.


After installing tcpdump, figure out on which physical network interface the HTTP requests get to your server. Next, figure the public IP-address of the machine you are working from - you need this to filter out unwanted noise if you run the connection test.

Let's call your server's IP-address A and your client's IP-address B.

Assuming the interface which holds the IP-address of this vhost is named eth0, you should run tcpdump as user root like this:

# tcpdump -n -s 1500 -w /tmp/tcpdump.pcap -i eth0 \(host A and host B and tcp port 80\) or udp port 53

(replace A and B with your IP-addresses; if you use https instead of http replace tcp port 80 with tcp port 443)

The program will start and will capture all traffic between your client machine and your server, as well as DNS packets (pretty much all of them I'm afraid, but we come back later to that) and store the content to the file /tmp/tcpdump.pcap.

Then, fire up your request. After you observed the problem, go back to the console and stop tcpdump by pressing CTRL+C.

Now I suggest you install the program Wireshark on your client machine for deeper analysis.

After installation, download the file from your server to your client machine and load the file in Wireshark through File -> Open.

To check if the TCP-handshake is OK, look for the first three packets. If capturing network traffic was OK, look for the first packet which has the TCP SYN flag set. Mark this using your mouse and choose Analyze -> Follow TCP Stream to filter out this single TCP-connection (we do this because usually you will have quite a lot TCP-connections from a single request to a webserver, but we only want to look into one for now).

A windows will open, just close it again, we don't need that. Now highlight the first packet and chose Edit -> Set time reference (Toggle). This will make all timestamps in the whole window relative to the first packet you marked so that we can have a look if timing is OK. Also set View -> Time Display Format -> Seconds since previous displayed packet if it's not the default.

Look at the first four packets. You will notice, they will have certain TCP-flags set:

  1. First packet from your client to the server has the SYN flag set.
  2. Second packet from your server to the client has the SYN and ACK flag set.
  3. Third packet from your client to the server only has the ACK flag set.
  4. The fourth packet now will have your first HTTP GET request.
  5. The fifth packet is the server ACKnowledging that the HTTP GET was received
  6. The sixt packet is the first data to arrive
  7. And this is even more data; if you look into the lower part of Wireshark you see the content.

Have a look at the timestamps of these seven packets. They all should follow very very close to each other; it should be looking something like this:

Wireshark screenshot 1

You can see all these packets are following very closely to each other, they are just hundreds of milliseconds apart. If this is the case for you, the TCP-connection works per se and we look more into DNS. But if this is not OK for you, if you see long delays or even TCP retransmissions or messages about "duplicate ACK" and the such, then you need to look into a deeper networking problem. Back to your VPS-provider then.

However, if this turns out to be OK, we look into DNS. For this we want to check about the following fact:

  • Does the DNS server respond in a timely manner for questions?
  • What exactly is the DNS question which is causing problems?

I will follow on the following hyptothesis:

  • The server sees your client's IP-address and wants to know it's DNS-name
  • For this it will send out a so called IN PTR question to the name server
  • I could imagine that this so called "reverese lookup" either fails or the nameserver answers to slowly.

So, now you have to do a little number-shuffeling. Assume your client's IP-address is 1.2.3.4. Now reverse the numbers (4.3.2.1) and add the suffix .in-addr.arpa to it. You've now built the correct query for Wireshark. In the Filter-field, say:

dns.qry.name == "4.3.2.1.in-addr.arpa"

In the example below I was asking for the full qualified name of my server at 78.47.204.196, so my filter is dns.qry.name == "196.204.47.78.in-addr.arpa":

Wireshark screenshot 2

In my case you see an interesting effect. My nameserver at 192.168.1.1 did not respond fast enough for my client's taste, so it sent out another request, although this time using IPv6. However, that doesn't really matter, it would just send out the same request again, if you don't have IPv6. You see that the real answer - yalla.hackers-r-us.org - comes 2.96 seconds after the request. It's slow, but the answer is there.

You know need to check the following things:

  • Does an answer to the request arrive at all?
  • How often is the same question issued over and over again?

If you see the request over and over sent out again in the course of 30 seconds, you know it's a DNS issue.

Sorry for the lengthy posting, but from what you wrote I assumed you're not too deep into these things.

share|improve this answer
    
Thanks for your help. Bounty expires in 24 hours so although I've not had chance to test your methods, you've obviously put in a great deal of work to helping me overcome my problem, so I'm inclined to offer it to you. –  Dan Hanly Mar 25 '13 at 13:11

Your instinct that it might be a DNS issue could well be right. On first connection, the VPS tries to reverse-resolve the incoming IP address for the logs, and if the DNS servers it's configured to use don't respond, it'll just sit there and wait for a while.

You could try changing the VPS's DNS servers to 8.8.8.8 and 8.8.4.4 (Google's open DNS resolvers) to see if that makes things any better. Alternatively, ask the VPS provider for its local DNS settings, and make sure they're set correctly in /etc/resolv.conf (or wherever those settings are in WHM/cPanel).

share|improve this answer
    
resolv.conf now contains: search static.as29550.net nameserver 8.8.8.8 nameserver 8.8.4.4 This is still giving the same results. Do I need to restart any services after this change? –  Dan Hanly Mar 14 '13 at 16:48
    
A couple of days after, this doesn't seem to have helped. I've changed Local DNS, my Router's DNS and the VPS's DNS to 8.8.8.8 and 8.8.4.4 –  Dan Hanly Mar 20 '13 at 11:20
    
@danielhanly.com if Flup's suggested problem is indeed the problem, then Flup's suggested solution|test does not exhaust the potential reasons for it. For instance, if your office's DNS server is broken, it won't matter which of Google's or your VPS provider's DNS resolvers query it. I would shell into the VPS and start with 'time dig -x ip-i-am-connecting-from'. –  Julian Fondren Mar 20 '13 at 21:38

If you were connecting over SSH to the host, I'd agree with the DNS suggestion. Howver, in this case, you're simply testing latency to serve an http request.

Looking at the first site in the chrome browser, I get latency info like:

2.04 s (onload: 2.04 s, DOMContentLoaded: 1.42 s)

The second site is:

4.59 s (onload: 4.40 s, DOMContentLoaded: 4.21 s)

Another tool I like to use is called check host.

Site 1:

http://check-host.net/check-report/1dc3fb

Site 2:

http://check-host.net/check-report/1dc3fa

Some other things right off the bat site 1:

www.google-analytics.com

Site 2:

https://twitter.com/statuses/user_timeline/spare_brain.json?callback=twitterCallback2&count=5

May I suggest also enrolling in the fantastic google webmaster tools:

https://www.google.com/webmasters/tools/home?hl=en

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.