AWS ELB with SSL backend adds proxy protocol inside SSL stream

Question

Note, this has been also posted in http://thread.gmane.org/gmane.comp.web.haproxy/27737

we are trying to configure this architecture:

• ELB terminating SSL, using preconfigured certificates. (this is a requirement because so only restricted people has access to the end user certs)
• ELB connects to HAproxy backend using SSL (also requirement)

• ELB sends proxy headers as described in http://amzn.to/1YajEG3

• HAproxy listens SSL in 443

• HAProxy is used for doing some HTTP transformations (modify header, etc).

Once ELB is configured as SSL+Proxy protocol, we tried to configure HAProxy by adding accept-proxy in the bind of the HTTPS frontend:

frontend https-in
    mode http
    # Note, I truncated this line because the maillist 80 chars limitations
    bind :443 accept-proxy ssl crt \
     /var/vcap/jobs/haproxy/config/cert.pem \
     no-sslv3 ciphers ...
    ...

But it fails: Received something which does not look like a PROXY protocol header.

Troubleshooting I found that ELB sends the PROXY header INSIDE of the SSL stream. For instance, I run openssl as a server:

$ openssl s_server -accept 443 -cert cert.pem
...

ACCEPT
bad gethostbyaddr
-----BEGIN SSL SESSION PARAMETERS-----
MFUCAQECAgMDBAIAnwQABDBsAWD78V/tz9KhYw4R/kpL5YPBxfF1qcmzxlclNDuz
0KWw9aGojVogjtBkH/zZOLWhBgIEVyoquqIEAgIBLKQGBAQBAAAA
-----END SSL SESSION PARAMETERS-----
Shared
ciphers:...
CIPHER is DHE-RSA-AES256-GCM-SHA384
Secure Renegotiation IS supported
PROXY TCP4 80.194.77.90 192.168.6.14 39220 443
GET / HTTP/1.1
User-Agent: curl/7.35.0
Host: something.com
Accept: */*

So I did a "chained" config in haproxy, one to do the SSL termination with pure TCP and the other to "extract" the proxy-protocol and do the HTTP transformations:

listen https-in
    mode tcp
    bind :443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem no-sslv3
ciphers ...
    server http 127.0.0.1:8081

frontend http-in-from-ssl
    mode http
    bind :8081 accept-proxy
    option httplog
    option forwardfor
    reqadd X-Forwarded-Proto:\ https
    default_backend http-routers

And that works!!!

So my questions are:

• Is this normal and expected? I cannot find any information about that.
• Is it possible to change the ELB behaviour to put the proxy-protocol header OUTSIDE of the SSL stream? I did not find any info about that.
• If not. Is it possible to change the behaviour of HAProxy to use one frontend but read the proxy-protocol header from inside the SSL stream?
• If not, is there a better way to 'chain' the config as I did above.

Thank you!

Answer 1

Here is the information for your question 1, check out the URL below.

http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/using-elb-listenerconfig-quickref.html

Loot at the last row of the table TCP/SSL Load Balancer (second table). It is just the case of yours. And it says explicitly

Does not support the Proxy Protocol header.

So for your question 2, the answer is no.

And I'm sorry I can't provide more help for your question 3 and 4. (Actually, based on my experience, for question 4, I think your way is good enough. Maybe my experience is not enough ;P )

Answer 2

I ran into this problem as well. However, I use nginx and not HA-proxy on my internal load balancers.

The solution is similar, but I figure it's worth posting:

#nginx.conf

user  nginx;
worker_processes  1;

error_log  /dev/stderr debug;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}

stream {
    upstream stream_backend {
        server 127.0.0.1:500;
    }

    server{
        listen 443 ssl;
        proxy_pass stream_backend;

        ssl_certificate /certs/local/public.crt;
        ssl_certificate_key /certs/local/private.key;
        ssl_protocols TLSv1.2;
        ssl_ciphers HIGH:!aNULL:!MD5;
    }
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '[$host] $remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /dev/stdout  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  60;

    #gzip  on;
    server {
        listen 8000;
        location /elb-status {
        keepalive_timeout 0;    # Disable HTTP keepalive

            access_log off;
            return 200;
        }
    }

    map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
    }

    server {
        listen 80 proxy_protocol;
        server_name test-nginx.corp.com;

        location / {
            keepalive_timeout 0;    # Disable HTTP keepalive
            return 301 https://$host$request_uri;
        }
    }

    upstream nginx-test-stack {
        server 10.42.111.6:80;
    }

    server {
        listen 127.0.0.1:500 proxy_protocol;
        server_name test-nginx.corp.com;
        real_ip_header proxy_protocol;

        location / {
            proxy_pass http://nginx-test-stack;
        }

    }
}

This allows me to enable E2E encryption for any TCP connections. If need be I can proxy websockets or https or just straight TCP