How to make Google Cloud Load Balancer respect the received X-Forwarded-Proto?

Question

My app structure uses GKE and CloudFlare. It looks like this:

CloudFlare -> GKE -> Ingress -> My app running nginx

I'm using the flexible SSL in CloudFlare, so only the connection between the user and CloudFlare uses HTTPS, all the remaining uses HTTP. I know CloudFlare sets the X-Forwarded-Proto to https in this situation, but when I see the headers my nginx app is receiving, it gets X-Forwarded-Proto: http.

I'm pretty sure this happens somewhere between GKE's Load Balancer and Ingress, as I can see that the CF-Visitor: {"scheme": "https"} header configured by CloudFlare is set to HTTPS. My understanding is that this means that CloudFlare did set X-Forwarded-Proto to https, but it got overwritten along the way.

Unfortunately, I couldn't get the header logs from the GKE Load Balancer (it seems they don't log the X-Forwarded-* headers at all), so I can't confirm 100% that CloudFlare is actually setting the headers, but I'd be pretty surprised if it isn't.

If that's true, Google Cloud is overwriting the X-Forwarded-Proto header with http. How can I avoid it doing so?

Edit: I have configured an nginx ingress instead of gce following https://cloud.google.com/community/tutorials/nginx-ingress-gke, and the X-Forwarded-Proto is set to https as expected. This is another signal that it's the gce ingress controller that is overwriting the X-Forwarded-Proto header.

Answer 1

As described in this article Cloudflare appends an X-Forwarded-Proto header which can either be HTTP or HTTPS depending on the protocol the user used to visit the site. If you believe the value of X-Forwarded-Proto should be maintained but was changed by GCLB, I'd recommend opening a feature request for this on Google issue tracker.

Answer 2

You can create a middleware:

# frozen_string_literal: true

require 'json'

class CloudflareProxy
  def initialize(app)
    @app = app
  end

  def call(env)
    return @app.call(env) unless env['HTTP_CF_VISITOR']

    env['HTTP_X_FORWARDED_PROTO'] = JSON.parse(env['HTTP_CF_VISITOR'])['scheme']
    @app.call(env)
  end
end

Use in config/application.rb:

config.middleware.use CloudflareProxy

A reference for CF-Visitor header:

https://support.cloudflare.com/hc/en-us/articles/200170986-How-does-Cloudflare-handle-HTTP-Request-headers-

CF-Visitor

A JSON object containing only one key called scheme. The value is identical to that of X-Forwarded-Proto (either HTTP or HTTPS). CF-Visitor is only relevant if using Flexible SSL.

Answer 3

Struggled with this on ingress-nginx too. After a couple of days of reading GitHub issues and GCP docs figured out the right values for helm chart:

controller:
  ingressClassResource:
    default: "true"
  service:
    loadBalancerIP: ${var.lb_ip}
    externalTrafficPolicy: Local
  config:
    enable-real-ip: "true"
    compute-full-forwarded-for: "true"
    use-forwarded-headers: "true"
    hsts: "false" # handled by cloudflare, we use flexible encryption mode 
    proxy-real-ip-cidr: "130.211.0.0/22,35.191.0.0/16,173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22"

And on the application level, you should use the second IP from the end in x-forwarded-for header. ExpressJS example:

const express = require('express');
const app = express();
// more on this here: https://expressjs.com/en/guide/behind-proxies.html
app.set('trust proxy', 2);
app.get('/', (req, res) => res.end(`
  ip: ${req.ip}
  headers: ${JSON.stringify(req.headers, null, 2)}
`));
app.listen(80, () => { console.log(`Server running at http://localhost:80`); });

In such case even when someone will try to pass fake ip via headers (or use VPN) your app will use the correct IP:

curl \
  --header "X-Forwarded-For: <FAKE_IP>" \
  --header "X-Forwarded-Proto: https" \
  --header "X-Forwarded-Host: example.com" https://example.com

  ip: <REAL_IP>
  headers: {
  "host": "example.com",
  "x-request-id": "...",
  "x-real-ip": "<REAL_IP>",
  "x-forwarded-for": "<FAKE_IP>,<REAL_IP>, <CLOUDFLARE_IP>",
  "x-forwarded-host": "example.com",
  "x-forwarded-port": "80",
  "x-forwarded-proto": "https",
  "x-forwarded-scheme": "https",
  "x-scheme": "https",
  "x-original-forwarded-for": "<FAKE_IP>,<REAL_IP>",
  "accept-encoding": "gzip",
  "cf-ray": "...",
  "cf-visitor": "{\"scheme\":\"https\"}",
  "user-agent": "curl/8.0.1",
  "accept": "*/*",
  "cf-connecting-ip": "<REAL_IP>",
  "cf-ipcountry": "...",
  "cdn-loop": "cloudflare"
}