Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have taken over a legacy web application that we cannot modify (the source code is broken and deployment fails) and we will eventually rewrite. Ideally we would migrate it one step at a time but this is not possible since we cannot effectively modify the application.

I am in charge of rewriting the application and there are some complex synchronization algorithms that I'd like to test against the data POST'd to the current API.

What's the easiest and safest way to capture incoming HTTP requests with all associated data? The solution must be transparent to API users. The server is running on Ubuntu Linux and we have SSH access to it. The web app is running on Apache 2 on Ruby.

  • Use some sort of packet sniffer to capture incoming traffic. I'd have to figure out a way to replay these requests against a server of my own.
  • Change the DNS to point to another IP that would capture and log the request data, then redirect the request to the production server. I don't think we have access to network infrastructure so I assume this must be executed as a web service that returns a request with a redirect header to the real server. Seems fragile and could pose security concerns for browsers?
  • Use some sort of Apache module to do this.

How do the above solutions compare in terms of:

  • The risk imposed on bringing the server and/or website down
  • The ease of implementation

Please feel free to suggest any further/better alternatives.

share|improve this question

3 Answers 3

up vote 1 down vote accepted

I would use tcpdump or ngrep, it would be nice to have the port on the switch connected to the web server mirrored, but in absence of that, you can run ngrep or tcpdump on the server itself.

You will need superuser access to run either of those programs.

You're going to want to read a bit, as you obviously know what you are looking for in the traffic, ngrep does allow you to select traffic by regex, which might allow you to pick out packets with better accuracy.

ngrep -l -q -d eth0 "^POST " tcp and port 80 -O dump.file

This would get you any HTTP data POSTed to port 80 on eth0. You might be able to pick out something much more specific. If you are going to be reading the traffic directly from the file, you might want to add -W byline as it makes the packets much more readable as it respects the line breaks, so you can see the packet written out more logically (for humans). -O dump.file will write the output of your packet capture to a file. The output can be as detailed as you'd like, to replay the packets, have a look at tcpreplay

share|improve this answer
    
Could you please comment on the output format of ngrep? I'll be trying to replay the requests on Windows. –  georgiosd Apr 8 '13 at 9:20
    
The output is as detailed as you'd like, the -W byline respects the line breaks, so you can see the packet written out more logically (for humans), to replay the packets, have a look at tcpreplay.synfin.net –  NickW Apr 8 '13 at 9:33
    
Please edit your original answer to include this comment for completeness. Thank you. –  georgiosd Apr 8 '13 at 20:48

The easiest and safest way to capture it would be to simply run a packet sniffer, as you suggested. If you simply do a web search for "replay http requests", you'll get any number of programs that can capture and then replay your requests. There are also professional tools for this, e.g. "LoadRunner" which is used at my job.

If you don't want to install any extra software, you could use tcpdump -s0 -w /path/to/output.file to capture the traffic. That will give you a file from which you can read, using "tcpdump -A -r /path/to/output.file', presumably using filters such as "dst=ip.to.webserver". But then you'd need something to replay it.

Here's an example of me using tcpdump on localhost:

$ tcpdump -i lo0 -s0 -w /tmp/tcpdump.out

meanwhile I access a web site on localhost from another window; once that's finished, I'll press ctrl-c to stop the tcpdump

$ sudo tcpdump -r /tmp/tcpdump.out  -t -q -n -A 'tcp dst port 80 and  (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
reading from file /tmp/tcpdump.out, link-type EN10MB (Ethernet)
IP 127.0.0.1.57499 > 127.0.0.1.http: tcp 122
E.....@.@.P............P.zE................
.9...9..GET /~dyho01/ HTTP/1.0
User-Agent: Wget/1.11.4 Red Hat modified
Accept: */*
Host: localhost
Connection: Keep-Alive

You could probably get the same information by opening the raw tcpdump file in a windows program such as wireshark.

share|improve this answer
    
Could you please comment on the output format of tcpdump? I'll be trying to replay the requests on Windows. –  georgiosd Apr 8 '13 at 9:19
1  
If you use '-A', the entire contents of the request will be written out. I'll add an example to my answer. –  Jenny D Apr 8 '13 at 9:27
    
Both answers are of good quality. I'll choose @NickW's answer because ngrep allows the use of convenient regular expressions. –  georgiosd Apr 8 '13 at 20:48

The packet sniffer option is a good one so long as the connection does not have encryption in play, then you will be SOL. In that case you will need a proxy model for the capture, where the users connect to the proxy using SSL, accepts the proxy certificate and then the connection to the proxy is again under SSL to the destination host. Testing tools like LoadRunner, Jmeter or the like would be good for capturing a proxy conversation for a single user to allow you to examine the structure of information being passed, but would be impractical for production use for multiple users.

I would look for a proxy server package where you could configure the proxy to dump the requests to a log for analysis purposes. This way you would have a raw view of the request outside of SSL and having to reassemble packets from a packet level trace.

share|improve this answer
    
Thanks for the proxy hint: that is a great solution when SSL/HTTPS is involved. –  Jeroen Wiert Pluimers Jul 17 at 9:00

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.