Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm about to pull out my hair on this one. I have a rails app hosted on Heroku, using Zerigo Free for DNS. The app is available at:

  • smartvark.com (preferred)
  • www.smartvark.com (some users insist on typing www, this redirects to remove the www)
  • smartvark.heroku.com (not given to users but perhaps useful for comparison in troubleshooting)

Users are intermittently (1 in 50 or so requests) experiencing extremely long load times (~2min) and when I try to triage by watching my server logs, their requests don't seem to hit until the end of the wait period. Typical load times for the site are fast around 200-400ms. I am using NewRelic and it isn't indicating any server load issues, although it picks up the end-user issue with its beacon and charts this time as "Network".

Using Firebug and Chrome devtools I am able to see the timeline when this happens on my machine, and they both show a long wait time before any response, which Firebug classifies as "DNS lookup" and Chrome doesn't seem to classify. After the first response happens, the rest of the site loads very fast.

share|improve this question
    
The problem seems to be worse on Macs, FYI. Can't see any reason for that, but the worst I get on Windows boxes is maybe 30s but Mac users see over 2min waits. –  SWrobel May 29 '11 at 19:50
1  
On my Macbook, it's quick, but that should have nothing to do with it. Notice you're using round robin DNS: is there a chance that one of those three hosts are slower than the other (not familiar with heroku.com)? i.e. I'd try rotating those 3 IP addresses in your hosts file, run some Firebug tests, and see if one or more is slower than the others. –  gravyface May 29 '11 at 20:16
    
It's very slow with Pingdon - tools.pingdom.com/fpt/?url=http://smartvark.com//… - You could give another DNS provider a go? I use PointHQ.com and they're great. Very fast and very easy to setup! –  Dean Perry May 29 '11 at 21:30
add comment

3 Answers

I'm going to guess that this is because your DNS provider (zerigo.net) is publishing IPV6 records for their DNS servers. Your Windows and MAC clients are using a DNS server that has IPV6 enabled, but doesn't have IPV6 connectivity. This causes a DNS timeout trying to access the DNS servers via IPV6 before failing back to IPV4. Trying turning off IPV6 on the DNS resolver and client machines and see if you get better results.

share|improve this answer
    
Sounds very reasonable. –  KCotreau May 29 '11 at 20:27
    
Interesting. How did you determine that Zerigo is publishing IPV6 records? Is there some way to look that up? All of the DNS tools I've played around with just seem to focus on IPV4. Also, do you have any clues how to solve this at a server level since I can't possibly have all of my users turn off IPV6? –  SWrobel May 29 '11 at 21:16
1  
Dig will show the AAAA records if you query with it. –  mfarver May 30 '11 at 0:09
add comment
up vote 0 down vote accepted

It turns out the problem was that one of the three round-robin IP addresses that Heroku assigned was bad. They figured this out in response to my support request. Thanks to everyone for your suggestions of methods and tools for chasing this down!

share|improve this answer
add comment

For others encountering this problem, this helped explain it:

http://tiwatson.com/blog/2011-2-17-heroku-no-longer-using-a-global-request-queue

Heroku doesn't use a global request queue, so a single long running request can backlock fast running requests.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.