Nginx ignoring HTTP headers before Host header when using proxy_pass to Apache

Question

When Nginx is sent the following request, Apache is unable to get the "app_version" HTTP header.

I normally use apache_request_headers() to get all headers. I have also examined the contents of $_SERVER

GET /stuff HTTP/1.0
app_version: 1.0
Host: example.com

However, when the following header is sent, apache_request_headers() does return the "app_version" HTTP header!

GET /stuff HTTP/1.0
Host: example.com
app_version: 1.0

The problem is, I am using a 3rd party REST client (Retrofit), and it seems to send the headers in a random order, so I cannot simply fix this by sending the Host header first. In any case, I don't see why a header with underscores in it, or the order it is placed in, should affect anything. (The header format is compatible with the HTTP spec). I would like a solution for the current problem, and not a workaround.

I have the following Nginx configuration:

underscores_in_headers on;

location / {
   proxy_set_header Host $host;
   proxy_set_header X-Forwarded-Proto $scheme;
   proxy_set_header X-Real-IP  $remote_addr;
   proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

   proxy_pass_request_headers on;
   proxy_pass http://127.0.0.1:8080;
}

What would be causing this inconsistency?

Answer 1

Try to add

proxy_set_header app_version $http_app_version;
underscores_in_headers on;

to the configuration.

It might be that only the underscores_in_headers setting is enough here.

Answer 2

The answer of Tero Kilkanen is correct. However you must add underscores_in_headers on; to the http section of the nginx conf, not only to your server section.

Nginx cannot know the virtual host to which the request is addressed before parsing the Host header. Therefore all headers that are parsed before the Host header are parsed in the "context" of the http configuration. If you don't have underscores_in_headers on; in the http configuration all headers that appear before Host and contain underscores are deemed invalid and are ignored. Once the Host header is parsed the configuration directives that belong to the server section corresponding to your request are activated. So if you have underscores_in_headers on; at server-level the headers appearing after Host and containing underscores are not discarded.

Answer 3

Same issue here, with a load balancer that is adding X-Forwarded-For, -Port and -Proto headers before the Host header:

$ nc -l 443
GET / HTTP/1.1
X-Forwarded-For: x.x.x.x
X-Forwarded-Port: 443
X-Forwarded-Proto: https
Host: y.y.y.y:443
Connection: keep-alive

Nginx just never fills $proxy_add_x_forwarded_for nor $http_x_forwarded_proto which means these are dropped.

Also, using curl to simulate the LB does result in nginx filling this in properly and there, the HOST comes before the x-fwd headers:

GET /admin/extra-commands/ HTTP/1.1
Host: x.x.x.x
X-Forwarded-For: 1.2.3.4

I think this is a limitation/bug of nginx