3

I'm about to pull out my hair on this one. I have a rails app hosted on Heroku, using Zerigo Free for DNS. The app is available at:

  • smartvark.com (preferred)
  • www.smartvark.com (some users insist on typing www, this redirects to remove the www)
  • smartvark.heroku.com (not given to users but perhaps useful for comparison in troubleshooting)

Users are intermittently (1 in 50 or so requests) experiencing extremely long load times (~2min) and when I try to triage by watching my server logs, their requests don't seem to hit until the end of the wait period. Typical load times for the site are fast around 200-400ms. I am using NewRelic and it isn't indicating any server load issues, although it picks up the end-user issue with its beacon and charts this time as "Network".

Using Firebug and Chrome devtools I am able to see the timeline when this happens on my machine, and they both show a long wait time before any response, which Firebug classifies as "DNS lookup" and Chrome doesn't seem to classify. After the first response happens, the rest of the site loads very fast.

3
  • The problem seems to be worse on Macs, FYI. Can't see any reason for that, but the worst I get on Windows boxes is maybe 30s but Mac users see over 2min waits.
    – swrobel
    May 29, 2011 at 19:50
  • 1
    On my Macbook, it's quick, but that should have nothing to do with it. Notice you're using round robin DNS: is there a chance that one of those three hosts are slower than the other (not familiar with heroku.com)? i.e. I'd try rotating those 3 IP addresses in your hosts file, run some Firebug tests, and see if one or more is slower than the others.
    – gravyface
    May 29, 2011 at 20:16
  • It's very slow with Pingdon - tools.pingdom.com/fpt/?url=http://smartvark.com//… - You could give another DNS provider a go? I use PointHQ.com and they're great. Very fast and very easy to setup!
    – user44923
    May 29, 2011 at 21:30

3 Answers 3

3

I'm going to guess that this is because your DNS provider (zerigo.net) is publishing IPV6 records for their DNS servers. Your Windows and MAC clients are using a DNS server that has IPV6 enabled, but doesn't have IPV6 connectivity. This causes a DNS timeout trying to access the DNS servers via IPV6 before failing back to IPV4. Trying turning off IPV6 on the DNS resolver and client machines and see if you get better results.

3
  • Sounds very reasonable.
    – KCotreau
    May 29, 2011 at 20:27
  • Interesting. How did you determine that Zerigo is publishing IPV6 records? Is there some way to look that up? All of the DNS tools I've played around with just seem to focus on IPV4. Also, do you have any clues how to solve this at a server level since I can't possibly have all of my users turn off IPV6?
    – swrobel
    May 29, 2011 at 21:16
  • 1
    Dig will show the AAAA records if you query with it.
    – mfarver
    May 30, 2011 at 0:09
0

It turns out the problem was that one of the three round-robin IP addresses that Heroku assigned was bad. They figured this out in response to my support request. Thanks to everyone for your suggestions of methods and tools for chasing this down!

0

For others encountering this problem, this helped explain it:

http://tiwatson.com/blog/2011-2-17-heroku-no-longer-using-a-global-request-queue

Heroku doesn't use a global request queue, so a single long running request can backlock fast running requests.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .