64

We have an application server that sometimes hangs. We suspect it is due to a bad request from a client.

Can nginx log the complete request/response (like fiddler captures) to files, so we can see the requests that were sent before the hang?

(We probably need to avoid pcap and that approach and do it all in nginx)

If nginx is not the right tool for this, what (other than a network analyzer) might be?

4
  • 2
    mitmproxy in reverse proxy mode should do what you are looking for. Oct 14, 2014 at 5:55
  • 1
    @VivekThomas this is an nginx question.... we are already using nginx and are not going to change. Dec 30, 2014 at 15:59
  • 3
    @samsmith Old question, but maybe this helps someone else: you don't have to give up nginx. Depending on the circumstances, you can just re-route nginx to another port temporarily, to allow mitmproxy to intercept the traffic and support the debugging. Then, once complete, you can just re-route nginx to the original port and shutdown mitmproxy. Nov 27, 2017 at 13:03
  • 2
    You can use modsecurity module, which can log full requests/responses, see nginx.com/blog/modsecurity-logging-and-debugging
    – Willem
    Oct 16, 2018 at 12:54

3 Answers 3

58

To get the request body sent by visitors, use client_body_in_file_only on; and log the "temporary" file it's written to in the logs by appending var $request_body_file to the log format. "Temporary" files will be located in client_temp directory by default.

You can log request headers $http_<header> too and sent headers with $sent_http_<header>.

If you have request body and headers you should be able to replay it and get the response your visitor had.

Also something like gor should highly be considered so you could replay the traffic on an other environment where you could let nginx write these temporary files without causing IO issues in production (nginx won't purge them with on value that's why It's not that "temporary" in this case).

10
  • 1
    @jwadsack Read the answer carefully. Jul 28, 2015 at 9:58
  • 4
    @XavierLucas I thought you were providing two different approaches. I didn't realize you were saying both client_body_in_file_only and $http_<header> would be needed. I get that now.
    – jwadsack
    Jul 28, 2015 at 17:31
  • 18
    Could you please share a more precise code?
    – Velkan
    May 9, 2017 at 13:40
  • 12
    Surely $http<header> is only useful if you know all the header names in advance
    – Ed Randall
    Feb 2, 2018 at 22:37
  • 12
    Can someone share an actual fragment of the nginx config, please?
    – Nowaker
    Jun 13, 2018 at 15:24
19

mitmproxy seems to be the right tool to do what you are asking.

mitmproxy is an interactive, SSL-capable man-in-the-middle proxy for HTTP with a console interface.

mitmdump is the command-line version of mitmproxy. Think tcpdump for HTTP.

Features

  • Intercept HTTP requests and responses and modify them on the fly.
  • Save complete HTTP conversations for later replay and analysis.
  • Replay the client-side of an HTTP conversations. Replay HTTP responses of a previously recorded server.
  • Reverse proxy mode to forward traffic to a specified server.
  • Transparent proxy mode on OSX and Linux.
  • Make scripted changes to HTTP traffic using Python.
  • SSL certificates for interception are generated on the fly.

The reverse proxy mode would let you capture the request and response just like Fiddler does.

1
  • In linux context, socat is also worth exploring for MITM. May 24, 2020 at 17:56
1

I was looking for an answer to this question, and found this: https://tarunlalwani.com/post/request-capturing-nginx-lua/

Basically, uses a lua script to gather and log all headers:

ngx.log(ngx.ERR, "REQUEST capturing started")
json = require("json")

function getval(v, def)
  if v == nil then
     return def
  end
  return v
end

local data = {request={}, response={}}

local req = data["request"]
local resp = data["response"]
req["host"] = ngx.var.host
req["uri"] = ngx.var.uri
req["headers"] = ngx.req.get_headers()
req["time"] = ngx.req.start_time()
req["method"] = ngx.req.get_method()
req["get_args"] = ngx.req.get_uri_args()


req["post_args"] = ngx.req.get_post_args()
req["body"] = ngx.var.request_body

content_type = getval(ngx.var.CONTENT_TYPE, "")


resp["headers"] = ngx.resp.get_headers()
resp["status"] = ngx.status
resp["duration"] = ngx.var.upstream_response_time
resp["time"] = ngx.now()
resp["body"] = ngx.var.response_body

ngx.log(ngx.CRIT, json.encode(data));

Then, refer to it from the nginx configuration:

location / {
    proxy_pass http://127.0.0.1:8081/;
    log_by_lua_file lua/request_logger.lua;
}

To gather the body, a block is needed in nginx main.conf:

#we must declare variables first, we cannot create vars in lua
set $response_body '';

body_filter_by_lua_block {
    -- arg[1] contains a chunk of response content
    local resp_body = string.sub(ngx.arg[1], 1, 1000)
    ngx.ctx.buffered = string.sub((ngx.ctx.buffered or "") .. resp_body, 1, 1000)
    -- arg[2] is true if this is the last chunk
    if ngx.arg[2] then
      ngx.var.response_body = ngx.ctx.buffered
    end
}

The post linked above details this a lot better.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.