2

Current situation:

I have a Triple-O OpenStack installation (Liberty) where even the "public" network is in a private environment (10.24.7.0/24 network). The only way to get access to that network is via a gateway host. I'd like to make Horizon accessible from the internet using a reverse proxy. Due to security concerns all outside communication should be encrypted via HTTPS. The reverse proxy should be used as SSL termination host and all inside traffic should be unencrypted.

The OpenStack deployment wasn't configured to use SSL/TLS and doesn't know that the "public" network isn't really public.

I'm currently trying to use Nginx as reverse proxy but usage of other reverse proxy software is also possible if they provide a solution to my problem.

The gateway host is running RHEL 7.2, Nginx is installed from official Nginx repo in version 1.11.1 (mainline).

gateway.example.com is the FQDN used in the examples,
1.2.3.4 is the external IP address of the gateway host used in the examples
10.24.7.9 is the "public" endpoint of the Triple-O installation

Work so far:

  • http://gateway.example.com loads the index.html where the certificate of the CA used for signing the server certificate can be downloaded.
  • Due to the fact that the OpenStack installation doesn't know that it is behind a SSL termination point I have to rewrite links in the content. This is done with two sub_filter directives, one generic (sub_filter 'http://$host' 'https://$host';) and one that replaces the OpenStack endpoint IP (sub_filter 'http://10.24.7.9' 'https://$host';)
  • The console feature of Horizon allows a user to connect to an instance's serial console via NoVNC embedded in Horizon. NoVNC uses websockets to establish a bi-directional TCP connection between the browser and the phyical host where the instance runs upon. To support this I had to configure Nginx to allow HTTP connection upgrades and to listen to port 6080.
  • iptables allows traffic on ports 80, 443 and 6080.

Problems:

  • On the Horizon page showing details to an instance instead of showing the body of a tab (overview, log, console, action log), the tab row is repeated and only the content of the overview tab is shown.repeating tabs
  • Connections to the console of an instance fail with error 1006. (The required URL to get the console show in its own browser tab was retrieved with nova get-vnc-console <ID> novnc and replacing the endpoint IP with the FQDN.) I suspect that this error is caused due to the fact that the connection that was upgraded from HTTP1.1 to websocket standard contains information regarding the "public" interface which cannot be accessed directly from the browser OR that the connection is refused by the NoVNC software due to a mismatch between the configured server address/hostname ("public") and the address in the request (proxy name/IP). Both of these aare speculation.NoVNC error

Nginx config file

ssl_certificate            /etc/nginx/certs/gateway.example.com.crt;  
ssl_certificate_key        /etc/nginx/certs/gateway.example.com.key;  
ssl_dhparam                /etc/nginx/certs/dh.pem;  
ssl_protocols              TLSv1.2 TLSv1.1;  
ssl_ciphers                AES256+EECDH:AES128+EECDH:!aNULL:!eNULL:!ECDSA:!SHA:!DSS;  
ssl_prefer_server_ciphers  on;  
ssl_session_cache          shared:SSL:10m;  
ssl_session_timeout        10m;  

server { # http  
  server_name                 gateway.example.com localhost 1.2.3.4;  
  listen                      *:80;  
  root                        /usr/share/nginx/html;  

  location / {  
    index                     index.html;  
  }  

  location ~ ^/dashboard {  
    return                    302     `https://$host$request_uri`;  
  }  

  location ~ ^/console {  
    return                    302     `https://$host:6080$request_uri`;  
  }  

  location ~ ^/websockify {  
    return                    302     `https://$host:6080$request_uri`;  
  }  
}  

server { # https  
  server_name                gateway.example.com localhost 1.2.3.4;  
  listen                     *:443;  
  ssl                        on;  

  location / {  
    sub_filter               '`http://10.24.7.9`'    '`https://$host`';  
    sub_filter               '`http://$host`'        '`https://$host`';  
    sub_filter_last_modified on;  
    sub_filter_once          off;  
    sub_filter_types         *;  
    proxy_pass               `http://10.24.7.9/$uri`;  
    proxy_request_buffering  off;  
    proxy_http_version       1.1;  
    proxy_set_header         Upgrade                           $http_upgrade;  
    proxy_set_header         Connection                        "upgrade";  
    proxy_set_header         Host                              $host;  
    proxy_set_header         Accept-Encoding                   "";  
    proxy_set_header         X-Real-IP                         $remote_addr;  
    proxy_set_header         X-Forwarded-Host                  $host;  
    proxy_set_header         X-Forwarded-Server                $host;  
    proxy_set_header         X-Forwarded-Proto                 $scheme;  
    proxy_set_header         X-Forwarded-For                   $proxy_add_x_forwarded_for;  
    proxy_connect_timeout    90;  
    proxy_send_timeout       90;  
    proxy_read_timeout       90;  
  }  
}  

server { # https on port 6080 for novnc  
  server_name                gateway.example.com localhost 1.2.3.4;  
  listen                     *:6080;  
  ssl                        on;  

  location / {  
    proxy_pass               `http://10.24.7.9:6080/$uri`;  
    proxy_request_buffering  off;  
    proxy_http_version       1.1;  
    proxy_set_header         Upgrade            $http_upgrade;  
    proxy_set_header         Connection         "upgrade";  
    proxy_set_header         Host               $host;  
    proxy_set_header         X-Real-IP          $remote_addr;  
    proxy_set_header         X-Forwarded-Host   $host;  
    proxy_set_header         X-Forwarded-Server $host;  
    proxy_set_header         X-Forwarded-Proto  $scheme;  
    proxy_set_header         X-Forwarded-For    $proxy_add_x_forwarded_for;  
 #    proxy_connect_timeout    90;  
 #    proxy_send_timeout       90;  
 #    proxy_read_timeout       90;  
  }  
}  

Does anyone have a working configuration for this kind of setup? I'd like to stick with Nginx but can make the change to Apache or any other software that can provide a solution.

Edit: clarified CA usage, added suspicions regarding NoVNC connection failure

3 Answers 3

0

I was able to solve the first issue as well. The reason for this odd behavior was caused because the reverse proxy dropped the query part of the request (everything behind the ?). This again was caused by misconfigured proxy_pass lines.

The solution to this was to remove /$uri from both proxy_pass lines in the configuration file (details see http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass ).

This is a working solution:

ssl_certificate /etc/nginx/certs/gateway.example.com.crt;  
ssl_certificate_key /etc/nginx/certs/gateway.example.com.key;  
ssl_dhparam /etc/nginx/certs/dh.pem;  
ssl_protocols TLSv1.2 TLSv1.1;  
ssl_ciphers AES256+EECDH:AES128+EECDH:!aNULL:!eNULL:!ECDSA:!SHA:!DSS;  
ssl_prefer_server_ciphers on;  
ssl_session_cache shared:SSL:10m;  
ssl_session_timeout 10m;  

server { # http  
  server_name gateway.example.com localhost 1.2.3.4;  
  listen *:80;  
  root /usr/share/nginx/html;  

  location / {  
    index index.html;  
  }  

  location ~ ^/dashboard {  
    return 302 `https://$host$request_uri`;  
  }  

  location ~ ^/console {  
    return 302 `https://$host:6080$request_uri`;  
  }  

  location ~ ^/websockify {  
    return 302 `https://$host:6080$request_uri`;  
  }  
}  

server { # https  
  server_name gateway.example.com localhost 1.2.3.4;  
  listen *:443;  
  ssl on;  

  location / {  
    sub_filter '`http://10.24.7.9`' '`https://$host`';  
    sub_filter '`http://$host`' '`https://$host`';  
    sub_filter_last_modified on;  
    sub_filter_once off;  
    sub_filter_types *;  
    proxy_pass `http://10.24.7.9`;  
    proxy_request_buffering off;  
    proxy_http_version 1.1;  
    proxy_set_header Upgrade $http_upgrade;  
    proxy_set_header Connection "upgrade";  
    proxy_set_header Host $host;  
    proxy_set_header Origin `http://$host`;  
    proxy_set_header Accept-Encoding "";  
    proxy_set_header X-Real-IP $remote_addr;  
    proxy_set_header X-Forwarded-Host $host;  
    proxy_set_header X-Forwarded-Server $host;  
    proxy_set_header X-Forwarded-Proto $scheme;  
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  
    proxy_connect_timeout 90;  
    proxy_send_timeout 90;  
    proxy_read_timeout 90;  
  }  
}  

server { # https on port 6080 for novnc  
  server_name gateway.example.com localhost 1.2.3.4;  
  listen *:6080;  
  ssl on;  

  location / {  
  proxy_pass `http://10.24.7.9:6080`;  
  proxy_request_buffering off;  
  proxy_http_version 1.1;  
  proxy_set_header Upgrade $http_upgrade;  
  proxy_set_header Connection "upgrade";  
  proxy_set_header Host $host;  
  proxy_set_header Origin `http://$host`;  
  proxy_set_header X-Real-IP $remote_addr;  
  proxy_set_header X-Forwarded-Host $host;  
  proxy_set_header X-Forwarded-Server $host;  
  proxy_set_header X-Forwarded-Proto $scheme;  
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  
  # proxy_connect_timeout 90;  
  # proxy_send_timeout 90;  
  # proxy_read_timeout 90;  
  }  
}  
0

I was able to solve the second of the issues (error 1006) by changing some of the configuration options. As the other problem still persists I cannot confirm that the NoVNC console works from within the Horizon UI but when I call the NoVNC URL directly I get a connection and can interact with the instance.

This is the working solution till now:

ssl_certificate /etc/nginx/certs/gateway.example.com.crt;  
ssl_certificate_key /etc/nginx/certs/gateway.example.com.key;   
ssl_dhparam                /etc/nginx/certs/dh.pem;  
ssl_protocols              TLSv1.2 TLSv1.1;  
ssl_ciphers                AES256+EECDH:AES128+EECDH:!aNULL:!eNULL:!ECDSA:!SHA:!DSS;  
ssl_prefer_server_ciphers  on;  
ssl_session_cache          shared:SSL:10m;  
ssl_session_timeout        10m;  

server { # http  
  server_name                 gateway.example.com localhost 1.2.3.4;  
  listen                      *:80;  
  root                        /usr/share/nginx/html;  

  location / {  
    index                     index.html;  
  }  

  location ~ ^/dashboard {  
    return                    302     `https://$host$request_uri`;  
  }  

  location ~ ^/console {  
    return                    302     `https://$host:6080$request_uri`;  
  }  

  location ~ ^/websockify {  
    return                    302     `https://$host:6080$request_uri`;  
  }  
}  

server { # https  
  server_name                gateway.example.com localhost 1.2.3.4;  
  listen                     *:443;  
  ssl                        on;  

  location / {  
    sub_filter               '`http://10.24.7.9`'    '`https://$host`';  
    sub_filter               '`http://$host`'        '`https://$host`';  
    sub_filter_last_modified on;  
    sub_filter_once          off;  
    sub_filter_types         *;  
    proxy_pass               `http://10.24.7.9/$uri`;  
    proxy_request_buffering  off;  
    proxy_http_version       1.1;  
    proxy_set_header         Upgrade                           $http_upgrade;  
    proxy_set_header         Connection                        "upgrade";  
    proxy_set_header         Host                              $host;  
    proxy_set_header         Origin                            `http://$host`;  
    proxy_set_header         Accept-Encoding                   "";  
    proxy_set_header         X-Real-IP                         $remote_addr;  
    proxy_set_header         X-Forwarded-Host                  $host;  
    proxy_set_header         X-Forwarded-Server                $host;  
    proxy_set_header         X-Forwarded-Proto                 $scheme;  
    proxy_set_header         X-Forwarded-For                   $proxy_add_x_forwarded_for;  
    proxy_connect_timeout    90;  
    proxy_send_timeout       90;  
    proxy_read_timeout       90;  
  }  
}  

server { # https on port 6080 for novnc  
  server_name                gateway.example.com localhost 1.2.3.4;  
  listen                     *:6080;  
  ssl                        on;  

  location / {  
    proxy_pass               `http://10.24.7.9:6080/$uri`;  
    proxy_request_buffering  off;  
    proxy_http_version       1.1;  
    proxy_set_header         Upgrade            $http_upgrade;  
    proxy_set_header         Connection         "upgrade";  
    proxy_set_header         Host               $host;  
    proxy_set_header         Origin             `http://$host`;  
    proxy_set_header         X-Real-IP          $remote_addr;  
    proxy_set_header         X-Forwarded-Host   $host;  
    proxy_set_header         X-Forwarded-Server $host;  
    proxy_set_header         X-Forwarded-Proto  $scheme;  
    proxy_set_header         X-Forwarded-For    $proxy_add_x_forwarded_for;  
 #    proxy_connect_timeout    90;  
 #    proxy_send_timeout       90;  
 #    proxy_read_timeout       90;  
  }  
}  

tl;dr

Changes:
In both server blocks for ports 443 and 6080 I added proxy_set_header Origin http://$host;. Don't know exactly what this is doing but it solved my problem.

This still leaves me with the first issue.

0

I stumbled upon this while I was struggling with this for a while too. And this really helped me out. However nginx doesn't like the ` quotes around the http://$host etc. When you remove those it works. This is a working configuration for me for example:

ssl_certificate /etc/ssl/cert.pem;
ssl_certificate_key /etc/ssl/cert.key;
ssl_ciphers HIGH:!RC4:!MD5:!aNULL:!eNULL:!EXP:!LOW:!MEDIUM;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_session_tickets off;

server { # http
  server_name openstack.example.com;
  listen *:80;

  location / {
    return 302 https://$host$request_uri;
  }

  location ~ ^/console {
    return 302 https://$host:6080$request_uri;
  }

  location ~ ^/websockify {
    return 302 https://$host:6080$request_uri;
  }
}

server { # https
  server_name openstack.example.com;
  listen *:443;
  ssl on;

  location / {
    sub_filter 'http://192.168.0.200:6080' 'https://$host:6080';
    sub_filter 'http://$host:6080' 'https://$host:6080';
    sub_filter_last_modified on;
    sub_filter_once off;
    sub_filter_types *;
    proxy_pass http://192.168.0.200;
    proxy_request_buffering off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_set_header Origin http://$host;
    proxy_set_header Accept-Encoding "";
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_connect_timeout 90;
    proxy_send_timeout 90;
    proxy_read_timeout 90;
  }
}

server { # https on port 6080 for novnc
  server_name openstack.example.com;
  listen *:6080;
  ssl on;

  location / {
    proxy_pass http://192.168.0.200:6080;
    proxy_request_buffering off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_set_header Origin http://$host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_connect_timeout 90;
    proxy_send_timeout 90;
    proxy_read_timeout 90;
  }
}

I am posting this because maybe someone is searching for an updated answer.

You must log in to answer this question.