Logging HTTPS traffic through a proxy

Question

How do I configure squid (or any other) proxy server to keep log of URLs visited, and files downloaded, over HTTPS? Is it possible?

Answer 1

You can do this only in very specific and specialised circumstances.

The problem is that the URLs visited, and exactly what went on in them, in an HTTPS connection is all protected by SSL. In order to look inside, you've got to break through that protection.

The only way to do that (short of becoming wildly famous by breaking a crypto algorithm or two) is to perform a man-in-the-middle attack on your users. You have to setup the proxy server with an SSL certificate for every domain in the world -- either a global wildcard certificate (one that has AltNames of *, *.*, *.*.*, and so on), or something that can generate SSL certificates on the fly (with your own local CA). You won't get a certificate like this from an established, "globally trusted" CA (well, maybe Comodo...), so you'll have to do everything self-signed, and then configure all devices that will use this proxy to trust that local CA. This trust issue is why you can only do this in "very specific and specialised circumstances".

Once you've got that setup, you can decrypt all of the HTTPS traffic as it comes through, log it, and then re-encrypt it using the end-site's own SSL certificate.

Before you jump up, click your heels and shout "yippee!", there are some things to note. The dangers and caveats in this process are considerable.

You need a proxy server (or farm) that's got enough grunt to terminate SSL for every connection that anyone who uses the proxy makes. For a small office, that's not too hard, but if you're working at any sizeable company you have a significant scaling problem ahead of you.

The security risks aren't trivial, either. This proxy farm of yours is now an incredibly high value target for anyone who wants to get their hands on some juicy credentials. Take note of the saying "don't put all your eggs in one basket" -- or more usefully in this case, "put all your eggs in one basket, just make sure it's a really strong basket". Don't think that people won't find out about it; it's not hard to identify these sorts of things if you know what you're looking for, and since you won't be able to deploy this in an organisation without people finding out (one way or another), anyone internal who might harbour naughty thoughts is likely to be very, very tempted. It might not be worth breaking into a dozen desktops to harvest credentials, but it's far more valuable -- and less work and risk -- to pop a central HTTPS proxy and watch everything go by.

As far as exactly how to do this in Squid or any other available proxy solution, rather than typing everything out by hand, I'll refer you to the Squid documentation for SSL bump and Dynamic SSL certificate generation which, together, will give you access to the encrypted traffic without generating undue browser security warnings. Once you've got that, logging the connections is trivial.