2

We have taken over a legacy web application that we cannot modify (the source code is broken and deployment fails) and we will eventually rewrite. Ideally we would migrate it one step at a time but this is not possible since we cannot effectively modify the application.

I am in charge of rewriting the application and there are some complex synchronization algorithms that I'd like to test against the data POST'd to the current API.

What's the easiest and safest way to capture incoming HTTP requests with all associated data? The solution must be transparent to API users. The server is running on Ubuntu Linux and we have SSH access to it. The web app is running on Apache 2 on Ruby.

  • Use some sort of packet sniffer to capture incoming traffic. I'd have to figure out a way to replay these requests against a server of my own.
  • Change the DNS to point to another IP that would capture and log the request data, then redirect the request to the production server. I don't think we have access to network infrastructure so I assume this must be executed as a web service that returns a request with a redirect header to the real server. Seems fragile and could pose security concerns for browsers?
  • Use some sort of Apache module to do this.

How do the above solutions compare in terms of:

  • The risk imposed on bringing the server and/or website down
  • The ease of implementation

Please feel free to suggest any further/better alternatives.

3 Answers 3

2

I would use tcpdump or ngrep, it would be nice to have the port on the switch connected to the web server mirrored, but in absence of that, you can run ngrep or tcpdump on the server itself.

You will need superuser access to run either of those programs.

You're going to want to read a bit, as you obviously know what you are looking for in the traffic, ngrep does allow you to select traffic by regex, which might allow you to pick out packets with better accuracy.

ngrep -l -q -d eth0 "^POST " tcp and port 80 -O dump.file

This would get you any HTTP data POSTed to port 80 on eth0. You might be able to pick out something much more specific. If you are going to be reading the traffic directly from the file, you might want to add -W byline as it makes the packets much more readable as it respects the line breaks, so you can see the packet written out more logically (for humans). -O dump.file will write the output of your packet capture to a file. The output can be as detailed as you'd like, to replay the packets, have a look at tcpreplay

3
  • Could you please comment on the output format of ngrep? I'll be trying to replay the requests on Windows.
    – georgiosd
    Apr 8, 2013 at 9:20
  • The output is as detailed as you'd like, the -W byline respects the line breaks, so you can see the packet written out more logically (for humans), to replay the packets, have a look at tcpreplay.synfin.net
    – NickW
    Apr 8, 2013 at 9:33
  • Please edit your original answer to include this comment for completeness. Thank you.
    – georgiosd
    Apr 8, 2013 at 20:48
1

The easiest and safest way to capture it would be to simply run a packet sniffer, as you suggested. If you simply do a web search for "replay http requests", you'll get any number of programs that can capture and then replay your requests. There are also professional tools for this, e.g. "LoadRunner" which is used at my job.

If you don't want to install any extra software, you could use tcpdump -s0 -w /path/to/output.file to capture the traffic. That will give you a file from which you can read, using "tcpdump -A -r /path/to/output.file', presumably using filters such as "dst=ip.to.webserver". But then you'd need something to replay it.

Here's an example of me using tcpdump on localhost:

$ tcpdump -i lo0 -s0 -w /tmp/tcpdump.out

meanwhile I access a web site on localhost from another window; once that's finished, I'll press ctrl-c to stop the tcpdump

$ sudo tcpdump -r /tmp/tcpdump.out  -t -q -n -A 'tcp dst port 80 and  (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
reading from file /tmp/tcpdump.out, link-type EN10MB (Ethernet)
IP 127.0.0.1.57499 > 127.0.0.1.http: tcp 122
E.....@[email protected]................
.9...9..GET /~dyho01/ HTTP/1.0
User-Agent: Wget/1.11.4 Red Hat modified
Accept: */*
Host: localhost
Connection: Keep-Alive

You could probably get the same information by opening the raw tcpdump file in a windows program such as wireshark.

3
  • Could you please comment on the output format of tcpdump? I'll be trying to replay the requests on Windows.
    – georgiosd
    Apr 8, 2013 at 9:19
  • 1
    If you use '-A', the entire contents of the request will be written out. I'll add an example to my answer.
    – Jenny D
    Apr 8, 2013 at 9:27
  • Both answers are of good quality. I'll choose @NickW's answer because ngrep allows the use of convenient regular expressions.
    – georgiosd
    Apr 8, 2013 at 20:48
1

The packet sniffer option is a good one so long as the connection does not have encryption in play, then you will be SOL. In that case you will need a proxy model for the capture, where the users connect to the proxy using SSL, accepts the proxy certificate and then the connection to the proxy is again under SSL to the destination host. Testing tools like LoadRunner, Jmeter or the like would be good for capturing a proxy conversation for a single user to allow you to examine the structure of information being passed, but would be impractical for production use for multiple users.

I would look for a proxy server package where you could configure the proxy to dump the requests to a log for analysis purposes. This way you would have a raw view of the request outside of SSL and having to reassemble packets from a packet level trace.

1
  • Thanks for the proxy hint: that is a great solution when SSL/HTTPS is involved. Jul 17, 2014 at 9:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .