up vote 1 down vote favorite
1
share [g+] share [fb]

I am an analyst programmer at my organisation and am finding some sort of intermitent time-out issue when using CVS and HTTP requests within our network.

After the time-out the request does complete though it takes just over 60-seconds, which is why I'm guessing that it's some sort of time-out fail-over problem happening.

I wish to try and figure out how to find if possible what the issue is, I'm assuming there's a bad rout being made somewhere or there's something wrong with one of the DNS servers. The infrastructure team has told me there isn't any issue with the network, which personally I'm thinking is a cop-out.

I have root access to two Linux (RHEL 5.4) machines.

Please excuse me if this task is obvious as I'm a software developer not a network engineer.

UPDATE

I thought I might mention that this problem occurs between clients and the CVS server and clients using VPN and the HTTP server. Our VPN clients do not reverse resolve and I've asked the network engineers to ammend that but they don't see that as being a problem.

link|improve this question
75% accept rate
Can you add some logging to your AJAX calls to get some timestamps of when things are being received, and when they are being returned? Is it possible whatever methods you are calling are actually timing out instead of the network? – mpeterson Nov 25 '09 at 14:48
Hi mpeterson, it doesn't appear to time out, it just takes 60 seconds for the request to complete, it's almost as if the packets are taking a very long time to get to the server. With the CVS server the server doesn't see the request until 60 seconds after I made it (using SSH), once the session is made it returns almost immediately. For the AJAX and other HTTP requests they are on a different server and do appear to time-out. I only have limited access to that server and it's windows which I know less about diagnosing proboems than on Linux. – Brett Ryan Nov 26 '09 at 2:50
feedback

2 Answers

up vote 1 down vote accepted

Often places will screw up their reverse records. You can tell you've got screwed up reverse records because if you run something like netstat -a and it takes a really long time to run and you get back a bunch of IP addresses in the rfc1918 address space. Not having reverse records in this space by itself isn't really a problem, but it is a problem if your DNS people forward their DNS requests to the providers or to a broken DNS server.

A quick way to verify if it is a DNS issue is to log onto the system and lookup an IP of someone connected to the system (look at netstat -a and look for established connections) and then run 

nslookup a.b.c.d (or whatever the IP of that host is)

if you've got an older system, you may need to type 

nslookup d.c.b.a.in-addr.arpa.

In either case, the result may be something like "can't find that address" but the answer needs to come back quickly. DNS timeouts can be on the order of seconds, and if you have 3 DNS servers in your resolv.conf, your server is going to try each one before it gives up. This can easily add up to a really annoying amount of time.

A quick way to illustrate the problem to your boss is to run netstat -an and then run netstat -a and then say "if our DNS was working properly, these would both run in almost exactly the same amount of time.

If it is a reverse-record issue, you can probably "fix" the problem by turning off reverse lookups in your applications. In this situation, it may be easier than getting another group involved.

There is also the remote possibility that there is a duplex mismatch between your servers and their switches. That can be tested by looking at the output of (windows) netstat -e or (unix) netstat -i. You're looking for "errors" or "collisions". If you see "collisions" then your end is mis-configured; it is half duplex and should be full duplex. If you see "errors" the switch end is half duplex and you're full duplex. Both counters should be zero, or at least small and not increasing. These problems can be really hard to track down because the link will work pretty well if it is unloaded and totally fall apart when there is lots of traffic.

link|improve this answer
Thankyou chris, very helpful and I will try to see if I can diagnose the latter part. I have seen this problem once before here which took me > 12 months to convince the infrastructure people there was a problem, it turned out that it was a misconfigured DNS server that no longer existed. I've tested the former part of your solution and everything seems fine, though VPN connections are not reverse resolving which could be compounding the issue. As mentioned I'm not much of a network person but I'll see if I can figure out the collisions side to your solution. – Brett Ryan Nov 26 '09 at 3:03
Sounds like you need better infrastructure people, or you need a supervisor who can pressure the infrastructure people's supervisor into actually making this work. Or, this isn't really a problem and you're just trying to make things work better (bravo to you) but noone else is complaining. If the developers are losing productivity over this, you need to make sure that mordac's boss gets an earfull. – chris Nov 26 '09 at 3:48
Thanks Chris, I have spoken to the IT manager who is now looking into it. It costs around 2 hours per day of developer time if they make 20 commits a day. It also costs time with our merchandiser supervisors using the web based applications time, though the problem is always blamed on their 3G air-cards without any actual backing/investigation. Our infrastructure team is small (3) with an attitude of "If I can't see the problem then there isn't one", and looking will only reveal a problem. – Brett Ryan Nov 26 '09 at 4:09
Thanks Chris, I'll mark that as answered I think. I'm sorry I haven't up-voted you but I can not as yet otherwise I would have done so. – Brett Ryan Nov 27 '09 at 4:54
Check the edit -- I'm going to describe exactly how to identify if it is a DNS issue. Rude and busy often look exactly the same from the outside so the more you can help them isolate the problem, the better... Just be careful with the politics of this sort of thing. – chris Nov 27 '09 at 5:21
show 2 more comments
feedback
up vote 1 down vote

If the request completes, then it's not a timeout issue. If it were a timeout issue the request would never complete, hence the name "timeout". Do you mean that some requests timeout and some complete after a long period of time, because that makes more sense than what you've stated in your post.

As far as tracking down the issue, there are a lot of areas to look at. Here's a few suggestions to get you started:

Run a tracert from a client machine to the server in question. Count how many hops it goes through. Each hop is a router of some sort. If the tracert goes directly from your client machine to the server, then there are no routers in the path.

Run a pathping from a client machine to the server in question and look for latency and packet loss between the two.

Install a packet sniffer on the server and start a capture. Submit a request from the client and look at the output of the packet sniffer on the server. If you see a siginificant delay between the request and reply in the sniffer output then it's a server issue. If there's no significant delay then it's a network issue.

link|improve this answer
Good heavens. That's going to be like trying to diagnose a problem by reading a 30mb tomcat dump. – chris Nov 25 '09 at 15:01
I don't see it being that difficult. Depending on the packet sniffer used, you can filter based on protocol and source and destination ip addresses so the actual dump you're looking at should be pretty small. IMHO a packet sniffer is an invaluable tool and is one of the first things I fire up when trying to diagnose network performance problems. Nothing but a packet sniffer can tell me what's happening at the packet level. – joeqwerty Nov 25 '09 at 15:12
Packet sniffers are invaluable, I'll agree. In situations where you're seeing performance problems, though, they won't directly see problems like mis-configured or slow DNS or duplex mismatches or systems with failing hard disks. I'd resort to the sniffer only after eliminating things that can be directly observed on the servers (dns, network configuration, is the app slow if I run a browser directly on the host, etc). – chris Nov 25 '09 at 15:50
@ chris: for the most part I agree with you but I would say that I like to start from the ground up. By running a capture on the server I can quickly (within 5 minutes) determine if the issue is on the server or the network by looking at the request\response packets in the capture. If I start by looking at the routers, switches, DNS, etc., etc. it could take hours to track down what might eventually turn out to be a server problem. – joeqwerty Nov 25 '09 at 16:05
@ chris: what I should have said is that I like to start at one end of the problem and work my way to the other. Since the server is the destination (one end) I would start my troubleshooting there and work my way to the client (other end). – joeqwerty Nov 25 '09 at 16:07
show 1 more comment
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.