Under narrow circumstances, I get a 100% reproducible delay of approximately 2 minutes when I request a public web page.
The page in question: https://id.openjdk.java.net/console/login
Splicing together client (curl) and server (apache) logs, a typical exchange looks like this:
[client] 15:42:03.694959 => Send header, 96 bytes (0x60) 0000: GET /console/login HTTP/1.1 <DELAY HERE> [server] [24/Feb/2015:22:44:26 +0000] "GET /console/login HTTP/1.1" [client] 15:44:25.984150 <= Recv header, 17 bytes (0x11) 0000: HTTP/1.1 200 OK
(Hours are different due to time zones, and seconds may be slightly off due to out-of-sync clocks.)
Things that are important:
- The client is connected via my home ISP, and given a public IP address in a particular subnet. Given a static IP address from a different range, the problem goes away.
- The '/console/login' page on this server is requested. There is no delay if I request 'console/forgotPassword'. I've not noticed a similar delay on any other server.
Things that are (apparently) not relevant:
- Home networking. Problem persists regardless of how the client connects to the ISP's modem, and when trying different modems.
- Time of day/congestion. The delay is consistent, no matter when the request occurs.
- DNS. Behavior is the same if I use an IP address rather than a host name.
- HTTPS handshaking. This portion of the 'curl' interaction always proceeds without delay; and there is no delay when accessing 'console/forgotPassword' via HTTPS.
- Client configuration. Reproducible in multiple browsers and via 'curl' on the command line. Reproducible on multiple client systems and OSes.
- Image/script/etc. loading. The delay is there when using 'curl', which requests no extra resources.
My question: what's the most plausible explanation for this? Or: what should I do next to diagnose it?
Server issues? As I understand the Apache log format, the delay is occurring before the server gets the GET request, but I'd appreciate it if someone could confirm that that's the proper way to read the log timestamps. If so, it's not clear to me what, if anything, the server might be doing before giving the request to httpd, or what httpd might be doing before getting a timestamp.
Networking issues? The tricky part here is that the delay apparently depends on the application-level message content: the specific page being requested. And that should be encrypted. The (small town) ISP admins say they perform no content filtering... I wonder, anyway, if some node along the way has my IP address range on a blacklist or something.