1

Under narrow circumstances, I get a 100% reproducible delay of approximately 2 minutes when I request a public web page.

The page in question: https://id.openjdk.java.net/console/login

Splicing together client (curl) and server (apache) logs, a typical exchange looks like this:

[client] 15:42:03.694959 => Send header, 96 bytes (0x60) 0000: GET /console/login HTTP/1.1
<DELAY HERE>
[server] [24/Feb/2015:22:44:26 +0000] "GET /console/login HTTP/1.1"
[client] 15:44:25.984150 <= Recv header, 17 bytes (0x11) 0000: HTTP/1.1 200 OK

(Hours are different due to time zones, and seconds may be slightly off due to out-of-sync clocks.)

Things that are important:

  • The client is connected via my home ISP, and given a public IP address in a particular subnet. Given a static IP address from a different range, the problem goes away.
  • The '/console/login' page on this server is requested. There is no delay if I request 'console/forgotPassword'. I've not noticed a similar delay on any other server.

Things that are (apparently) not relevant:

  • Home networking. Problem persists regardless of how the client connects to the ISP's modem, and when trying different modems.
  • Time of day/congestion. The delay is consistent, no matter when the request occurs.
  • DNS. Behavior is the same if I use an IP address rather than a host name.
  • HTTPS handshaking. This portion of the 'curl' interaction always proceeds without delay; and there is no delay when accessing 'console/forgotPassword' via HTTPS.
  • Client configuration. Reproducible in multiple browsers and via 'curl' on the command line. Reproducible on multiple client systems and OSes.
  • Image/script/etc. loading. The delay is there when using 'curl', which requests no extra resources.

My question: what's the most plausible explanation for this? Or: what should I do next to diagnose it?

Server issues? As I understand the Apache log format, the delay is occurring before the server gets the GET request, but I'd appreciate it if someone could confirm that that's the proper way to read the log timestamps. If so, it's not clear to me what, if anything, the server might be doing before giving the request to httpd, or what httpd might be doing before getting a timestamp.

Networking issues? The tricky part here is that the delay apparently depends on the application-level message content: the specific page being requested. And that should be encrypted. The (small town) ISP admins say they perform no content filtering... I wonder, anyway, if some node along the way has my IP address range on a blacklist or something.

  • Any chance of doing a packet capture on the client to see exactly what's going on at the network layer? – jotap Mar 4 '15 at 18:45
  • Are the server logs from the actual system hosting that service, or from another copy of the service? The Apache log won't be written until after Apache's gotten a response from whatever is serving the application code (in this case it appears to be a Tomcat service), so the suspect that I'd point to first would be what's going on in the JVM. – Shane Madden Mar 4 '15 at 19:06
  • 1
    Broken/slow DNS reverse lookup for the ip(s) you originate from, preferably from the server where the delay comes from ? (host -t PTR <ip you appear to be coming from>). Hypothesis, for the slow url(s), apache tries to do a reverse-lookup of your ip, the reverse lookup is either slow and/or times out, request appears to be slow. – Kjetil Joergensen May 24 '16 at 0:05
  • I've had an experience like this because (believe it or not) peering. The ISP of the server refused to peer with the ISP of the client network, so all the traffic got routed around the whole globe despite the physical distance between the server and the client being like 10 km. – Broco Dec 4 at 12:10
0

I had a similar issue, which appears to have been caused by a timeout of the xdebug remote connection. If your page is using php with xdebug, perhaps that is causing your issue.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.