2

I have an environment consisting of four servers networked together. One server acts as the server, and the other three act as clients for running automated tests and Linux benchmarking using Phoromatic.

The four systems are all behind a corporate firewall. If I set the "http_proxy" and "https_proxy" environment variables on the clients, they can connect to the outside world and download tests and such, however they will not connect to the server as they try to connect to the local server using the proxy. Since I wanted to cache the package downloads, tests, etc... I set up a Squid proxy on the server system, and configured it as a transparent proxy, but it only works with http requests.

What I'd like to do is have the http requests handled via the cache, and forwarded to the parent proxy as needed. Obviously I can't decrypt the ssl sessions, but I can't figure out how to have the Squid proxy forward https requests to the parent proxy. Additionally, the squid proxy is running on the same box as the Phoromatic server, which is Web based but uses a user-configurable nonstandard port, but Squid likes to block requests to said port, even when it's added to the configuration as being allowed.

I would be OK with just having the clients use the corporate firewall directly for https and ftp requests and either just using the Squid cache for http requests, or ditch the Squid proxy altogether and have the clients set to not use the proxy for local hosts.

It's really frustrating me, since most of the time I'm great at hunting down information and making things work on my own, without having to pick anyone else's brain about it, but I guess I have a rather unique situation! And yes, I have tried the Phoronix forum for Phoromatic to no avail.

Servers are SuperMicro X8DTT dual chassis systems running Fedora 24. Network configuration consists of a GbE connection to a switch (used as the connection to the outside world) as well as two 10Gb on each system, also connected through a switch, but the 10Gb system is not connected to the outside world - they're used for bandwidth testing (The drivers for the 10Gb cards is what the system is set up to test)

  • What I ended up doing was I set the server system to talk to the proxy. On the client systems, I set the proxy in the configuration for DNF, but left it empty for the Phoromatic client. The client figures out it can't talk to the outside world, and has the server pull the requested content, since it's configured to talk to the proxy. (The underlying problem was the software wasn't smart enough to figure out that it only needed the proxy to talk to the outside world, and didn't need it to talk to the server.) So I may just forgo the caching proxy since the time savings is likely not worth it... – Andrew Bowers Feb 18 '16 at 20:37
2

I'll be short (yeah, it doesn't look short like at all, but otherwise it would be way longer and totally unreadable).

  • it doesn't look like you need proxy. like totally.
  • in a modern environment a cache ratio can be between zero and 40% (and I'm judging based on my proxies byte ratios), so if you want to save that amount of data, you can of course, use proxy. But consider this: in today's enterprise environment the role of a proxy is more of authorizing users on their way to the WAN access, than caching data. And that's the main reason for doubting your choice.
  • if you still need the proxy, it doesn't mean you have to decrypt the HTTPS. just let it live. it won't be cached, so what. It's in its design.
  • if you're still insisting on decrypting HTTPS - you can use sslBump technique. But this may be illegal in some countries, and furthermore this complicates things a lot. Like A LOT. I advise you not to go this way only for caching purposes.
  • don't serve local traffic via proxy: it adds latency, it loads the proxy, with unnecessary traffic (since it's cheap and LAN channels are way wider than WAN), it complicates the debugging and it adds parasite network dependencies, so it's unwise.
  • since I doubt you need proxy, I doubt even more you need a parent proxy. Looks like you're just having this thing .... you know, being into proxies. use one if you need it.
  • may be instead of the proxy you just need a fast and decent web-server, line nginx. So in a situation when your web-servers get overloaded it can act as a balancer for a farm with a l2-cache.
  • squid isn't scaling that well. for 10-gigs bandwidth you'll have to use SMP squid features, and this have it's downsides. Like unbalanced load on squid workers, SMP issues in squid internals, and so on. It may be solvable if you have previous experience with squid, but unlikely if you have set it up like for the first time.
  • finally, if you're deciding to stick with squid, it doesn't have to be transparent: you can configure WPAD for clients, and let servers decide how they should acces Internet.
1

I set up a Squid proxy on the server system, and configured it as a transparent proxy, but it only works with http requests.

This is expected as HTTP and HTTPS work differently and can't be dealt with in the same way by a proxy. When an HTTPS request is redirected to a proxy port transparently, the proxy can not look into the encrypted traffic, and therefore can't handle it. A transparent proxy actually works more like a "Man in the Middle" that interecetps a http traffic without the users knowing it, which is possible because of the lack of security in http protocol.

https://en.wikipedia.org/wiki/HTTPS

HTTPS (also called HTTP over TLS,[1][2] HTTP over SSL,[3] and HTTP Secure[4][5]) is a protocol for secure communication over a computer network which is widely used on the Internet. HTTPS consists of communication over Hypertext Transfer Protocol (HTTP) within a connection encrypted by Transport Layer Security or its predecessor, Secure Sockets Layer. The main motivation for HTTPS is authentication of the visited website and protection of the privacy and integrity of the exchanged data.

In its popular deployment on the internet, HTTPS provides authentication of the website and associated web server with which one is communicating, which protects against man-in-the-middle attacks. Additionally, it provides bidirectional encryption of communications between a client and server, which protects against eavesdropping and tampering with and/or forging the contents of the communication.[6] In practice, this provides a reasonable guarantee that one is communicating with precisely the website that one intended to communicate with (as opposed to an impostor), as well as ensuring that the contents of communications between the user and site cannot be read or forged by any third party.

As you can see, HTTPS is supposed to protect against man in the middle attack and not allow it. The following Squid page explains all your questions and confusions in details.

http://wiki.squid-cache.org/Features/HTTPS

When a browser comes across an https:// URL, it does one of two things:

  • opens an SSL/TLS connection directly to the origin server or

  • opens a TCP tunnel through Squid to the origin server using the CONNECT request method.

Squid interaction with these two traffic types is discussed below.

CONNECT tunnel

The CONNECT method is a way to tunnel any kind of connection through an HTTP proxy. By default, the proxy establishes a TCP connection to the specified server, responds with an HTTP 200 (Connection Established) response, and then shovels packets back and forth between the client and the server, without understanding or interpreting the tunnelled traffic. For the gory details on tunnelling and the CONNECT method, please see RFC 2817 and the expired Tunneling TCP based protocols through Web proxy servers draft.

CONNECT tunnel through Squid

When a browser establishes a CONNECT tunnel through Squid, Access Controls are able to control CONNECT requests, but only limited information is available. For example, many common parts of the request URL do not exist in a CONNECT request:

  • the URL scheme or protocol (e.g., http://, https://, ftp://, voip://, itunes://, or telnet://),

  • the URL path (e.g., /index.html or /secure/images/),

  • and query string (e.g. ?a=b&c=d)

With HTTPS, the above parts are present in encapsulated HTTP requests that flow through the tunnel, but Squid does not have access to those encrypted messages. Other tunnelled protocols may not even use HTTP messages and URLs (e.g., telnet).

When a browser is configured to use a proxy manually, then it uses the CONNECT method mentioned above and it works. Having said that, there are ways to configure transparent proxy to intercept https traffic (ssl-bump), which is not leagal and not recommended and must be used with caution.

Bumping CONNECT tunnels

{X} WARNING: {X} HTTPS was designed to give users an expectation of privacy and security. Decrypting HTTPS tunnels without user consent or knowledge may violate ethical norms and may be illegal in your jurisdiction. Squid decryption features described here and elsewhere are designed for deployment with user consent or, at the very least, in environments where decryption without consent is legal. These features also illustrate why users should be careful with trusting HTTPS connections and why the weakest link in the chain of HTTPS protections is rather fragile. Decrypting HTTPS tunnels constitutes a man-in-the-middle attack from the overall network security point of view. Attack tools are an equivalent of an atomic bomb in real world: Make sure you understand what you are doing and that your decision makers have enough information to make wise choices.

Squid SslBump and associated features can be used to decrypt HTTPS CONNECT tunnels while they pass through a Squid proxy. This allows dealing with tunnelled HTTP messages as if they were regular HTTP messages, including applying detailed access controls and performing content adaptation (e.g., check request bodies for information leaks and check responses for viruses). Configuration mistakes, Squid bugs, and malicious attacks may lead to unencrypted messages escaping Squid boundaries.

From the browser point of view, encapsulated messages are not sent to a proxy. Thus, general interception limitations, such as inability to authenticate individual embedded requests, apply here as well.

  • I know the proxy won't work with https, my intended meaning of "it only works with http requests" was that it's only set up to handle http requests. What I wanted it to do was cache http requests (so that if, say, all three clients requested the same Linux update package, it'd only have to go to the Interwebz for the first request) and have https, etc.. get forwarded to the upstream proxy. But I actually got it figured out shortly after I posted this, see my comment on the original post. – Andrew Bowers Feb 18 '16 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.