0

I'm really new to managing servers, and I've discovered that I can run ngrep and see connections from Google to our site (which is currently getting thousands upon thousands of 5xx errors and "crawl anomaly" errors in search console).

My question is how can I see if the connections are successful (if google is actually receiving a 200 response code, or not). Also, does Connection: close. mean that the connection was closed and they did not receive a response?

Example of output from ngrep.

$ sudo ngrep -il -d eth0 -W byline "Chrome/41.0.2272.96 Mobile Safari/537.36" port 80

GET / HTTP/1.1.
Host: subdomain.com.example.com.
User-Agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) X-Middleton/1.
Accept: text/html,application/xhtml+xml,application/signed-exchange;v=b3,application/xml;q=0.9,*/*;q=0.8.
Amp-Cache-Transform: google;v="1..3".
Cookie: BLAH.
From: googlebot(at)googlebot.com.
If-Modified-Since: Mon, 04 May 2020 15:51:07 GMT.
X-Forwarded-For: IP, IP.
X-Middleton: 1.
X-Middleton-Ip: IP.
X-Real-Ip: IP.
X-Snipe: BLAH.
Accept-Encoding: gzip.
Connection: close.
2
  • Already tried with tcpdump?
    – AndreaCi
    May 6, 2020 at 17:47
  • No I haven't tried tcpdump. Could you provide an example command with tcpdump to show only connections that are 200 response code, and conversely a command for 5xx response codes? I need to see which connections are "failing" to response with the actual web pages.
    – NAMAssist
    May 6, 2020 at 18:00

1 Answer 1

2

Apache Logging

As per @hermanb response on the other question you asked on this topic, the easiest way to do this is not via tcpdump or ngrep, rather its via simply grepping the apache logs - which record the information you want in an easier to read form, including the response code.

The log file location and layout will vary depending on your setup and are very customizeable. The default logs will likely have the information you want, but if not, you can create your own logs.

On my server there are lines

 LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
 CustomLog ${APACHE_LOG_DIR}/other_vhosts_access.log vhost_combined

In the config file which creates a file /var/log/apache/other_vhosts_access.log. The "CustomLog" bit at the front says to use a custom log format, and the vhost_combined bit at the end describes the log format.

You can, of-course, have more then 1 log file, and more then 1 log format. This is all described at https://httpd.apache.org/docs/2.4/mod/mod_log_config.html#customlog

Once you have a format you can - in real time, grep the log file to see what you are looking for - for example,

tail -f /var/log/apache2/other_vhosts_access.log | grep --line-buffered " 200

"

would look at each line of the log file as its created and look for " 200 ". and using "grep -v" to invert the search. There are lots of more complex ways to get information out, buy using more complex grep expressions and/or more/other logging.

You should also read the page linked above as its got lots of information on formatting your own logs - for example maybe try

 LogFormat "%s %v:%p/%r" statuscodelog
 CustomLog /path/to/mylogfile.log statuscodelog

Which should, in theory, produce a simple output in the fom

200 example.com/requested/path1 500 example.com/requested/file/causes/error

You can then load this into a spreadsheet for easy analysis or grep this more specifically and simply by something like

tail -f /path/to/mylogfile.log | grep --line-buffered "^5"

To get a list of URLS returning status code 5** Of-course, this can be tailored to catch whatever you want, in whatever format you decide to create. This is just a fairly minimal example.

Connection Close

This is a duplicate of What does "Connection: close" mean when used in the response message? - the accepted answer shows that the "Connection: close" is something the web server or client sends to show the connection should not be persisted. The link goes into more depth, but it is simply following the specification for an HTTP/1.1 connection for applications that do not support persistent connections.

2
  • Ok, this worked for me: tail -f /var/log/nginx/access.log | grep --line-buffered "Googlebot" - I can see Googlebot accesses being logged. My next problem is that it's not recording subdomains so I can't see which subdomains are responding with "200" status. I can't see any information on logging the subdomain for the access log access either in the provided urls?
    – NAMAssist
    May 7, 2020 at 6:45
  • I've figured out to write my own custom logs including the subdomain. Thank you - I should be able to start finding the 500 errors soon, hopefully! I'll either update this or start a new Q (if it's different enough to warrant a new question).
    – NAMAssist
    May 7, 2020 at 7:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .