I've recently inherited an infrastructure where all the users go through an AWS ELB -> nginx proxy -> AWS ELB -> Apache. The problem I'm seeing is that the client's IP is showing twice in Apache, a line similar to this:

"x.x.x.x, x.x.x.x" - - [26/Oct/2016:18:59:10 +0000] "GET /....

nginx's access log for this virtual host is fine and dandy:

x.x.x.x - - [26/Oct/2016:18:59:10 +0000] "GET /

The problem is that I'm not really sure how to debug where this gets lost. I assume it's either the headers nginx is sending or something weird in Apache, although I suspect it is the X-Forwarded-For headers that nginx is passing. Here's the relevant virtual host configuration for an affected domain:

location / { if ($xxx-example-com-maintenance) { return 503; } if ($http_origin ~* (https?://xxx\.example\.com)) { add_header 'Access-Control-Allow-Origin' "$http_origin"; } proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass $upstream;

Now, Apache's log format looks like this:

LogFormat "\"%{X-Forwarded-For}i\" %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" SESSION-%{frontend}C %h" forwarded

The latter was copied from the previous web stack and worked just fine. Kinda stuck here because it all looks good from my perspective. I did find nginx real_ip_header and X-Forwarded-For seems wrong however I'm running nginx 1.10. Apache version is 2.4.23. Any insights on how to troubleshoot this further would be appreciated. Thanks in advance.


It turns out that X-Forwarded-For header is meant to behave like that and the solution was to use a different header that the ELB wasn't using - in my case X-Real-IP. I'm now able to see a single IP in the logs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.