nginx filtering headers with a dot in the header name, even after enabling ignore_invalid_headers

Question

A client (developed by a third party) is sending a request with headers like foo.meta-digest (note the "dot" in the header name). My nginx reverse proxy is removing these headers from the request even if I added ignore_invalid_headers off; (see ​http://nginx.org/r/ignore_invalid_headers) to the server { ... } block for this vhost:

upstream backend {
   keepalive 128;
   keepalive_requests 1000;
   keepalive_timeout 120s;
   server 127.0.0.1:32000 max_fails=0 fail_timeout=10;
}

server {

   listen 443 ssl http2;
   server_name foo.com;
   root /var/www/empty;
   ssl_certificate /etc/nginx/certs/foo.com.crt;
   ssl_certificate_key /etc/nginx/certs/foo.com.key;

   ignore_invalid_headers off;

   location / {
       proxy_pass http://backend;
       proxy_set_header        Host              $http_host;
       proxy_set_header        X-Real-IP         $remote_addr;
       proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header        X-Forwarded-Proto $scheme;
       proxy_set_header        Connection      "";
   }
}

I cannot reproduce this problem with any other client, except for this specific one and I don't have access to the source code of this client, it's proprietary software.

If I send the very same request using curl or any other web client, all the headers, including the ones with a "dot" in the name, are passed to the backend as they should.

I tried even running a mitm attack against the nginx host and intercepted the network traffic from the client. The client is definitely sending the foo.meta-digest header as expected. Somehow nginx is filtering the header, but only for this specific client.

What else can I do to debug this problem?

Answer 1

Just had a similar problem, and I only fixed it by adding the ignore_invalid_headers off also to the default server

The client in my case was doing two weird (and bad-practice) things:

1. It was connecting in TLSv1.2, but without using the SNI extensions.
   This is used by the client to give (in cleartext) the domain name it is connecting to, so that the server is able to use the right certificate for the right vhost.
   Without this, the client always gets the certificate for the default vhost.
   ...which the client had pinned, so that it was accepted even though it was invalid for the domain.

Note that this only affects the TLS certs you will get from the server! In theory (see point 2 later) nginx will still use the Host: header to determine which is the correct vhost, and which conf to use.

Note that unless you are pinning the certificate in the client application, and unless you are sure about the certificate, this is a very unsafe thing to do. You would be opening yourself to MITM attacks.
Also, SNI was developed like 12 years ago, and it is necessary for virtual hosting.

2. Nginx will use the Host: Header to know which vhost configuration to apply.
   As you can guess since I told you to add the ignore_invalid_headers in the default server, this does not work on a special case:
   RFC7230, section 5.4 says that the Host header SHOULD be the first thing you send.
   In my case (and proably yours) it wasn't.
   The headers before the Host Headers are thus parsed as in the default server configuration, then nginx finds the Host and switches to the correct vhost.

Weird, almost non-standard, definitely bad practice, but not technically wrong.

Hope it helps