A production server has a constant stream of incoming data which is delivered over HTTP. To test a new revision of the system, I deploy the new version to a staging server, run tests on it, and when I'm satisfied, I want to try it on the real data, but without affecting the production system.

To achieve that, I would like to copy the incoming HTTP traffic and direct the copy to the staging system. I had previously baked that into the application itself, but I don't want to do that anymore, for a number of reasons. What are my options?

  • Are there proxies that do this?
  • Are there web servers that do this? The closest I found was Apache's forensic module, but I'm not sure if that's suitable, it clearly has a slightly different goal.
  • Or should I do it at the Internet layer?

Of course, the staging server wants to send replies somewhere, just like the production system. And naturally, these replies shouldn't go back to the production system's clients. Simply discarding them at some stage is OK, but it would be great if I could direct them to some logging facility for inspection.

  • since HTTP traffic uses TCP and TCP is a 2-way negotiation, forking the traffic wont work because then the forking proxy server would have to figure out how to handle the response packets in a sensible manner... – djangofan Nov 26 '09 at 18:20
  • @djangofan - That's sort of what I was referring to in my last paragraph. Since I'm looking at the topmost layer here, I wouldn't care about what happens on the TCP layer, the proxy could just open a new connection and bin bucket everything that comes back. – Hanno Fietz Nov 26 '09 at 20:10

I'd be inclined to capture the TCP traffic, rip the request data out of that, and use a few lines of script to feed the requests back into the staging server. For requests that you don't expect to change between staging and production, you could even have the responses from the TCP stream compared to those produced by the staging server, to validate correctness.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.