TL;DR - I'm planning on using an application-level firewall to forward only requests for HTML content to a transparent proxy. So the initial GET request for google.com (for example) that returns an HTML file will be forwarded to Squid (the transparent proxy), but all subsequent requests for static assets (CSS, JS, image files, etc) won't go through the proxy.

Can you think of any way this might impact the user session?

Some background:

We have a router onsite, but want to do content filtering using a transparent proxy + ICAP server hosted on AWS. Initially we planned on forwarding all port 80 traffic to the transparent proxy, but were worried about the extra 50-100ms we'd incur for each request as it was routed through our EC2 instance, and more importantly would run up unacceptable bandwidth costs. Recently we discovered that the router's firewall can filter using deep packet inspection (so could match outgoing requests against an "Accept text/html.*" header for example).

At first blush this seems like a much more manageable solution, but my networking experience is limited and wanted to surface any "gotchas" and/or examples of how this is obviously a horrible idea before going too far down this path.


Transparent HTTP proxying typically has little to no impact to user sessions in my experience.

Your strategy and what you actually intend to accomplsih seems unclear to me, though. I think the reality of how HTTP works is going to present a problem for you. Perhaps you can talk more about what you're really trying to accomplish from an end-result perspective. I'm not saying it's a "horrible idea" but it's completely unclear what you hope to gain.

From an HTTP protocol perspective your strategy has a major issue. Your layer 7 firewall can't know, by definition, the MIME type of a resource being requested until after the request is made. You can do a match on the filename (seeing it if ends in ".html", etc) but any URL can return an arbitrary MIME type. This .b0rk file is text/html, but a match rule in your firewall on .html or .htm wouldn't "know" that. The request has to be made to the remote server and the remote server has to respond before the MIME type is known.

Why do you assume that other types of files (CSS, Javascript, images, etc) are "static". They certainly don't have to be. The object to which any URL refers can be "dynamic".

If you're worried about bandwidth costs why not just host a transparent proxy locally? A local proxy wouldn't incur a tenth of a second latency for requests to hit the proxy, and you wouldn't have the potential bandwidth costs associated with shipping all of your data out from a third-party datacenter. You'd also see some improvement in your local bandwidth utilization when the proxy caches objects and you could let it cache everything (that it cna).

I've run a 1,000 seat educational network through a Squid-cache instance running on hardware that, today, would seem pitiful. Unless you're talking about a very, very large network a very modest machine could handle the load for you. If you're worried about a single point of failure you could look at using Web Cache Communication Protocol (WCCP), if your edge device supports it, to do failover between a couple of caches.

If you're looking at doing this to scan HTML for "malicious" code then I think you're looking at it backwards. I'd be more worried about malicious code in Javascript and application/octet-stream objects than about text/html objects. I am dubious of scanning, anyway, because anything can be sufficiently obfuscated as to slip by scanning (see the Halting problem). Unless you're going to to SSL interception you're also going to miss anything delivered over HTTPS, too. Scanning can catch known exploits and I do see it as a valid part of an IT security architecture, but 0-day is nearly always going to slip right by.

  • Thanks for the response! Sorry the goal wasn't clear - our use case is a little specific so I didn't want to confuse the issue. Due to the region in which the application will be deployed we'll have limited hardware options and can't host the transparent proxy locally. The ultimate goal is to understand the type of material being requested - broader than blocking a specific domain like denying requests to "facebook.com" for example. I was thinking about looking for "text/html" in the outgoing "Accept" request header to isolate HTML requests. – jonathon Jul 10 '13 at 20:11
  • Using the "Accept:" header doesn't buy you anything, because it almost always going to include a wildcard. The browser doesn't know what it's looking for any more than your proxy would. If you're just looking at getting the URLs being accessed (and possibly the MIME type returned) you'd be better off using an edge device that is capable of logging that. (Again, w/o SSL interception you'll be losing some quantity of data.) – Evan Anderson Jul 10 '13 at 20:40
  • @jonathon Sounds like you just need an extended Wireshark session. – Michael Hampton Jul 10 '13 at 23:07
  • I'd think that if his edge device can filter on HTTP headers it ought to be able to send URLs over SYSLOG or something similar. – Evan Anderson Jul 10 '13 at 23:24
  • Thanks for the logging ideas guys. In an ideal scenario we'd be able to take action in real time -- like recognizing disallowed content prior to fulfilling the user's request and redirecting, manipulating the document to inject a warning message, etc. Good point about the wildcard in the Accept headers. I need to do some more testing there, but it looks like only requests expecting HTML include "text/html" explicitly, so matching against that string might be conclusive. Anyway - I really appreciate all the feedback. – jonathon Jul 10 '13 at 23:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.