nginx: Log complete request / response with all headers?

Question

We have an application server that sometimes hangs. We suspect it is due to a bad request from a client.

Can nginx log the complete request/response (like fiddler captures) to files, so we can see the requests that were sent before the hang?

(We probably need to avoid pcap and that approach and do it all in nginx)

If nginx is not the right tool for this, what (other than a network analyzer) might be?

Answer 1

To get the request body sent by visitors, use client_body_in_file_only on; and log the "temporary" file it's written to in the logs by appending var $request_body_file to the log format. "Temporary" files will be located in client_temp directory by default.

You can log request headers $http_<header> too and sent headers with $sent_http_<header>.

If you have request body and headers you should be able to replay it and get the response your visitor had.

Also something like gor should highly be considered so you could replay the traffic on an other environment where you could let nginx write these temporary files without causing IO issues in production (nginx won't purge them with on value that's why It's not that "temporary" in this case).

Answer 2

mitmproxy seems to be the right tool to do what you are asking.

mitmproxy is an interactive, SSL-capable man-in-the-middle proxy for HTTP with a console interface.

mitmdump is the command-line version of mitmproxy. Think tcpdump for HTTP.

Features

• Intercept HTTP requests and responses and modify them on the fly.
• Save complete HTTP conversations for later replay and analysis.
• Replay the client-side of an HTTP conversations. Replay HTTP responses of a previously recorded server.
• Reverse proxy mode to forward traffic to a specified server.
• Transparent proxy mode on OSX and Linux.
• Make scripted changes to HTTP traffic using Python.
• SSL certificates for interception are generated on the fly.

The reverse proxy mode would let you capture the request and response just like Fiddler does.

Answer 3

I was looking for an answer to this question, and found this:

Basically, uses a lua script to gather and log all headers:

ngx.log(ngx.ERR, "REQUEST capturing started")
json = require("json")

function getval(v, def)
  if v == nil then
     return def
  end
  return v
end

local data = {request={}, response={}}

local req = data["request"]
local resp = data["response"]
req["host"] = ngx.var.host
req["uri"] = ngx.var.uri
req["headers"] = ngx.req.get_headers()
req["time"] = ngx.req.start_time()
req["method"] = ngx.req.get_method()
req["get_args"] = ngx.req.get_uri_args()


req["post_args"] = ngx.req.get_post_args()
req["body"] = ngx.var.request_body

content_type = getval(ngx.var.CONTENT_TYPE, "")


resp["headers"] = ngx.resp.get_headers()
resp["status"] = ngx.status
resp["duration"] = ngx.var.upstream_response_time
resp["time"] = ngx.now()
resp["body"] = ngx.var.response_body

ngx.log(ngx.CRIT, json.encode(data));

Then, refer to it from the nginx configuration:

location / {
    proxy_pass http://127.0.0.1:8081/;
    log_by_lua_file lua/request_logger.lua;
}

To gather the body, a block is needed in nginx main.conf:

#we must declare variables first, we cannot create vars in lua
set $response_body '';

body_filter_by_lua_block {
    -- arg[1] contains a chunk of response content
    local resp_body = string.sub(ngx.arg[1], 1, 1000)
    ngx.ctx.buffered = string.sub((ngx.ctx.buffered or "") .. resp_body, 1, 1000)
    -- arg[2] is true if this is the last chunk
    if ngx.arg[2] then
      ngx.var.response_body = ngx.ctx.buffered
    end
}

The post linked above details this a lot better.