0

A website which has a link to my website recently changed from http to https. Before that, a hit from its link appeared in my apache access_log, as expected - but that is no longer true. I now see hits generated by the webpage load, but not the initial webpage hit itself. Testing that website's link myself confirms this behavior - the initial hit is not logged.

What can explain this? How can I again get the initial hit logged (I use its referer field to keep track of accesses from that website)? In the log I do see referers from other https sites - but not from this particular website. FYI my apache configuration is pretty standard. Nothing has changed on my end and hits from other sites appear as expected. Trying to get more info using "LogLevel debug" did give anything I found useful.

Jack

PS: I'm assuming the change to https is the cause, since that is the obvious change - but perhaps the website has made additional changes I'm not aware of. Also, I do have modsecurity installed, but cannot see anything in its logs relevant to this.

1

Referrer information is never sent in this case. Browsers will refuse to send the Referer header to an HTTP URL when the referring site is HTTPS.

If you want to get the referrer information again, you will need to move your site to HTTPS, and also get the referring site to update their link so that it points to HTTPS rather than HTTP.

  • this is very useful information, but the question asks (or at least appers to) why an access request is missing from the log. The fact that he wants to use the Referer header means he will also want it supplied, but your answer does not explain why the entry is apparently missing from the log. – Rob Lambden Aug 14 at 7:15
  • @RobLambden Either one of us has misinterpreted the question, or the OP has misspoke. It's extremely unlikely that he is actually missing a log entry, and most likely that he actually has the entry and that he simply missed it because it didn't have the referer he was expecting. – Michael Hampton Aug 14 at 15:38
  • yes, I've never heard of a situation where log entries 'go missing' but Apache does support log filters and requests can be cached ... – Rob Lambden Aug 14 at 16:00
  • Hopefully my (badly formatted) comment to Rob Lambden below showing truncated log entries and what I consider the "missing" initial hit better illustrates the situation. – user48297 Aug 14 at 16:23
  • Very useful info - as one running only a single website, admittedly reluctant to go through hassle of change to https since the website does not ask for identifying info, even to post a comment – user48297 Aug 14 at 16:26
0

Your question isn't very clear ... but I think I understand your problem.

You say that you have checked the access frm the other website. Load up their page in a web browser, open up the developer tools and Network information. Then click your link. You should see that a request is generated and you should be able to see both the request and response - both the headers and the actual request.

From that information you should be able to tell what has changed. It's possible that the link no longer goes directly to your website in the way you expect it to but instead is going through a tracking mechanism. In any event you should be able to see the request, response and all headers.

Unless you have specifically disabled something or are implementing some kind of log filtering you should see all requests that reach your server in your log. As you have not changed this you should expect it to still be there. My guess is that it's not what you think it is.

In some browser configurations a cached web page can be served without checking with the original server. This is another posibility, but you should see that the response is cached in your network information view.


From the additional information you have posted in comments you can see that the resources required for your page are being loaded, and are being loaded by the page you exepct (from the Referere field) but you have no log of the initial request for that page.

Try making the page request wth cache disabled (again use the netowrk view of the developer tools tab to disable cache). You can disable caching of that page if you can set the correct headers, however you cannot force clients to honour that, but you should reasonably expect the end client to honour Cache-Control: headers. Ideally this should be set at the server as a header, not using a meta tag.

As per Michael's answer you may also have a problem if you need the Referer tag to be present.

  • In Chrome, I opened the referring website and activated developer tools Network pane. But clicking on the link opened my webpage in a different tab - and by the time I could open developer tools Network pane there, earlier info did not appear so not helpful. – user48297 Aug 13 at 17:42
  • I know caching may be a problem. Am now wondering if what I am not seeing could be caused by combination of (1) caching by my own and other browsers and (2) the now-https website now removing referer information. (1) could explain why I am getting log entries showing the "linked" webpage in "Referer" fields without any preceeding "URL requested" field entry. (2) could explain why I never see any referers from the referring website, even though there should be considerable traffic coming from it to my site. – user48297 Aug 13 at 17:43
  • Can you confirm that the https website could be opting to not have referer info passed on to my website? Personally I don't know how that could be done, but I imagine it might be possible. – user48297 Aug 13 at 17:44
  • You can make a request without caching. It's not clear to me what you mean by a preceeding URL Requested. Whay don't you post your log (anonymise if necessary) to the question and explain what doesn't make sense. – Rob Lambden Aug 13 at 20:20
  • Using elipses to reduce character count, I see lines ala ... "GET /lib/js/CONTEXTMENU/ContextMenu-fork-jack.js HTTP/1.1" 200 1057 "http://bigsurtrailmap.net/route_metrics.html" ... ... "GET /cgi-bin/get_shortest_trailroute.cgi ... HTTP/1.1" 200 21890 "http://bigsurtrailmap.net/route_metrics.html" ... but do not see a preceding line for the initial webpage causing those hits (as I normally do for other accesses) ie ... "GET /route_metrics.html ... HTTP/1.1" " ... – user48297 Aug 14 at 0:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.