Log incoming requests on Ubuntu (ports 80, 443)

Question

We have Tomcat running on Ubuntu server. It runs a web service, open to the internet.

Sometimes it has a sudden spike of traffic and goes down. There is nothing unusual in Tomcat access logs. I guess it is because some of the requests are so 'heavy' that they never finish and hence are not recorded to Tomcat access logs.

Is there a way to configure Ubuntu or a 3rd party tool to log incoming requests in the following format (below)?

Date, Time, URL (with query string params), IP address (of client)

There should be one line per request. Each request should be logged before it is executed. Only incoming requests to ports 80 and 443 should be logged.

Answer 1

2 possible solutions for you.

1. Configure squid or similar to act as a reverse proxy, look up 'squid reverse proxy'. caveat: You may find you have compatibility issues, but i'm no expert.
2. It is possible to use iptables to log traffic coming into your server. See the LOG target in the iptables man-page. caveat: iptables will NOT be http aware. You will get all the information you want except the requested url.

Answer 2

Install nginx (or Apache) and configure it as a simple reverse proxy in front of Tomcat. Below is a sample configuration you can use in a virtual host in nginx. The default access log format should give you the information you need.

server {
  listen 80;
  server_name www.example.org example.org;

  location /context/ {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://tomcat-server:8080/context/;
  }

  access_log /var/log/nginx/www.example.org-access.log;
  error_log /var/log/nginx/www.example.org-error.log;
}

Edit: Even if Tomcat was unable to serve the request, it should be logged in nginx (and also in Tomcat).