I have backend web servers that receive requests by way of haproxy->nginx->fastcgi. The web app used to see multiple ip's coming through in the X-Forwarded-For header, chained together with commas (most original IP on the left).

At some point in the recent past (just noticed, so not sure what caused it) something changed, and now I'm only seeing a single IP passed in the header to my web application.

I've tried with haproxy 1.4.21 and 1.4.22 (recent upgrade) with the same behavior. Haproxy has the forwardfor header set:

option forwardfor

Nginx fastcgi_params config defines this header to be passed to the app:

fastcgi_param HTTP_X_FORWARDED_FOR $http_x_forwarded_for;

Anyone have any ideas on what might be going wrong here?

EDIT: I just started logging the $http_x_forwarded_for variable in nginx logs, and nginx is only ever seeing a single IP, which shouldn't ever be the case, as we should always see our haproxy ip added in there, right? So, issue must either be in nginx handling of the variable coming in, or haproxy not building it properly. I'll keep digging...

EDIT #2: I enabled request and response header logging in HAProxy, and it is not spitting anything out for X-Forwarded-For, which seems very odd:

Oct 10 10:49:01 newark-lb1 haproxy[19989]: [10/Oct/2012:10:49:01.467] http service/newark2 0/0/0/16/40 301 574 - - ---- 4/4/3/0/0 0/0 {} {} "GET /2zi HTTP/1.1" O

Here are the options i set for this in my frontend:

mode http
option httplog
capture request header X-Forwarded-For len 25
capture response header X-Forwarded-For len 25
option httpclose
option forwardfor

EDIT #3: It really seems like haproxy is munging the header and just passing on a single one to the backend. This is fairly impacting to our production service, so if anyone has an ideas it would be greatly appreciated. I'm stumped... :(

migrated from stackoverflow.com Oct 11 '12 at 0:25

This question came from our site for professional and enthusiast programmers.


To respond to your last question in the comment, it is normal to have more than one IP address in XFF, this header is a list of values, and proxies often add their client's address there. Since everyone in the long chain appends values there, your server must use them in reverse order. For instance, the last value will be the one added by the haproxy instance in front of the server, and the previous value will be the one added by the reverse-cache before haproxy, etc...

If you'd prefer not to adapt the application to correctly parse the header, you can also ask haproxy to remove it before adding its own XFF header:

reqidel ^X-Forwarded-For:

That way the server will only get the value added by haproxy which will be haproxy's client.

  • Willy, the problem is that I used to ser multiple addresses in the header and wrote code in my application to pull out the first non-private IP in the chain and use it, which worked great. A some point something changed and now I only ever receive a single address in the header at my application level, which seems like haproxy is applying that reqidel logic by default or something. Do you see any reason that would be overwriting the chain instead of appending, based on my config? – JesseP Nov 1 '12 at 15:35
  • It does not do that by default, and I suspect that your application does not iterate over all headers but randomly picks one and uses the value it carries. Haproxy never modifies the existing header, it always adds one with the new value. It may be the one your application is seeing, but your application should also see the previous ones. – Willy Tarreau Nov 9 '12 at 7:35

I think there is some confusion in the way you attempt to use the X-Forwarded-For header.

First, the fact that nginx sees one IP address means that haproxy correctly adds it. The header contains only the source address that haproxy received the connection from, so it is normal that you don't see haproxy's IP address in nginx logs.

Second, it is also expected that you don't observe x-forwarded-for in incoming requests, because only some outgoing proxies add the header, but in general it's recommended not to do so when going to the internet. If some users send you a request with such a header, you'll see it in haproxy's capture, and nginx will log both this value and the client's IP added by haproxy.

What I don't understand is your point #3, because you seem to assume that the header is necessarily present in incoming requests, which obviously is not the case, judging by both haproxy's captures and nginx logs. I have just sent you a request right now with "X-Forwarded-For: Hi,Jesse,this is Willy" that you should see in both haproxy and nginx logs if it can help you troubleshoot.

What is possible is that earlier you were used to see multiple addresses there because either one of your main visitors was using and outgoing proxy which added the XFF header, or because you had another reverse-proxy in front of haproxy (eg: apache, stunnel, ...).

BTW, you should replace "option httpclose" with "option http-server-close", it will enable keep-alive with the clients and reduce the page load time for those who experience a high latency.

  • Willy, thanks for the response. while I did have something in front of haproxy at one point, I am pretty sure that haproxy frequently sent on XFF headers with a chain of IPs in the past. Did this used to happen in older versions, where the newer versions just send on the just one IP? At some point i wrote code in my app to pull the proper IP from the XFF chain, instead of the request IP, if the XFF header was present (at least my haproxy in the request path) so i didn't geo-target the user to the location of my load balancer. – JesseP Oct 14 '12 at 10:40
  • Is there a better way (at least with >= haproxy 1.4.21) to determine the clients IP for geolocation purposes (not their private IP incase they are running through a proxy)? – JesseP Oct 14 '12 at 10:41
  • Also,thanks for the suggestion around keep alive. I'll be sure to give that a shot for our website and client dashboard to see how it improves load times. :). Or service,where we use geoip most, is usually just single requests, so we chose to disable keep alive there. – JesseP Oct 14 '12 at 10:42
  • I don't get your point then. The client's IP is sent in the XFF header to the server. Concerning keep-alive, you may then use "forceclose" to disable it more efficiently than with "httpclose" as it releases the server connection earlier. – Willy Tarreau Oct 14 '12 at 11:12
  • Is there a way to put the clients public ip in the header instead of their private, if they are passing through a proxy on the way? I see a good handful of 10., 192., etc (private ips) in the header the today, which I'm of course unable to geo-locate. – JesseP Oct 14 '12 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.