1

I currently have squid setup to use a self-signed certificate for MITM to cache HTTPS requests. This works. If an item is not in the cache I want to request from an online proxy like Crawlera. Unfortunately Crawlera only offer an http endpoint. When I try to forward to this endpoint, everything works for HTTP, but for HTTPS I received the error: Handshake with SSL server failed: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

I'm using squid 4.2. Is there a way I can configure squid so I can specify it as a proxy for an https request and then have it act as a cache or forward to an HTTP proxy (that supports CONNECT)? If at some point I'm transmitting in plain text it doesn't matter at all for this application.

The following is my configuration for Squid:

http_port 3128 ssl-bump \
  cert=/apps/server_crt.pem key=/apps/server_key.pem \
  generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
sslcrtd_program /apps/squid/libexec/security_file_certgen -s /apps/squid/var/lib/ssl_db -M 4MB sslcrtd_children 8 startup=1 idle=1 
acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump bump all
acl localnet src 10.0.0.0/8     # RFC1918 possible internal network
acl localnet src 172.16.0.0/12  # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7       # RFC 4193 local private network range
acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl Safe_ports port 1025-65535  # unregistered ports
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
coredump_dir /apps/squid/var/cache
maximum_object_size 10 GB
cache_dir ufs /apps/squid/var/cache/squid 100 16 256
cache_mem 256 MB
maximum_object_size_in_memory 512 KB
cache_replacement_policy heap LFUDA
range_offset_limit -1
quick_abort_min -1 KB
offline_mode on
http_access allow localnet
http_access allow localhost
http_access deny all
refresh_pattern . 525600 100% 525600 ignore-reload ignore-no-store ignore-private ignore-auth ignore-must-revalidate store-stale

cache_peer proxy.crawlera.com parent 8010 0 ssl login=APIKEY:
never_direct allow all

Update

If I change the ssl_bump directives above to the following:

acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3

ssl_bump stare step2
ssl_bump bump step3

Request for HTTPS resources tunnel all the way through both proxies to the target and correctly return the response to the caller, but it no longer has MITM access at the Squid proxy to cache the results, so they CONNECT though to Crawlera on subsequent requests for the same resource. HTTP on the other hand will go through both proxies if it's not in the cache, otherwise it does get returned from the cache.

This is still not the solution I'm looking for though, I would like to cache HTTPS as well.

Improve this question

2 Answers 2

Reset to default
4

As far as I can tell, you should simply set the cache_peer to http (remove ssl option, which by the way, if you plan to use it, should be tls since squid v4.x)

cache_peer proxy.crawlera.com parent 8010 0 ssl login=APIKEY

Try:

cache_peer proxy.crawlera.com parent 8010 0 login=APIKEY

PS: I just noticed the : at the end of the that line after APIKEY, I'm not sure that's correct, you might want to remove it, unless it has something to do with the APIKEY itself and was a typo.

Improve this answer
5
  • Thanks Leo, I removed the : and I also tried removing the ssl/tls flag and it made no difference unfortunately. Previously I was trying to do this with the latest Squid 3.5 and the error message I got when there was no ssl flag was: assertion failed: PeerConnector.cc:116: >>> "peer->use_ssl". I believe there might be an even older version of squid (~3.2) that allowed an HTTPS request going to an HTTP proxy, then it was disabled for it's poor security implications (with the use_ssl message). Now it seems it just assumes an ssl peer, flag or not, if the request was originally HTTPS.
    – LaserJesus
    Sep 20, 2018 at 16:51
  • I think removing the : didn't cause any errors because https was never making it to the proxy authentication request and http was already cached. I recently made some more progress and I had to put the : back in to avoid getting a 407 Proxy Authentication error (the proxy just uses an API key for authentication, not username and password, so maybe it needs this funky approach)
    – LaserJesus
    Sep 20, 2018 at 17:12
  • I've added an update with some more info. Still not a solution but perhaps some extra clues...
    – LaserJesus
    Sep 20, 2018 at 17:41
  • Hello LaserJesus, I don't think the HTTP option is deprecated, and as I have never used crawlera or other public proxies, I really can't tell about their support. But I found that you might use HTTPS with Crawlera using their certificate, here is the doc: support.scrapinghub.com/support/solutions/articles/…
    – Leo Gallego
    Sep 21, 2018 at 2:44
  • Thanks again Leo, I think Crawlera have an HTTP proxy but you can tunnel through it to an HTTPS endpoint (using the CONNECT verb). I'm not sure exactly how it all works since you do need to accept a certificate from Crawlera and they are able to change the user agent header, so they must be doing MITM too. That said, the problem I was experiencing is actually an issue with Squid itself whereby it doesn't work when you combine ssl_bump (MITM style) and cache_peer. I found a solution to this though and I've described it in my answer here.
    – LaserJesus
    Sep 21, 2018 at 20:58
2

So, it turns out that what I was experiencing is a known issue with Squid. Alex Rousskov on the Squid Users mailing list was nice enough to help me out and point me in the direction of a fork of Squid by Measurement Factory. On that fork is a branch of Squid that can actually deal with this issue of combining ssl_bump with a cache_peer. https://github.com/measurement-factory/squid/tree/SQUID-360-peering-for-SslBump

I was able to build from this branch and then my original configuration worked!

I don't think any of the more recent branches from the Measurement Factory fork include this fix. It's also not official Squid Project code so it won't be supported by SP. But I was told MF are woking on getting it included in an official SP release and MF certainly have precedent for successfully doing that.

Here's my Dockerized end to end solution https://github.com/brett--anderson/squid_proxy

Improve this answer
1
  • Could you please provide a www link to the related squid-users mailing list messages?
    – 把友情留在无盐
    Jan 3, 2020 at 11:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .