While trying to figure out why our Varnish 4.1 install (on CentOS7 via the varnish-cache.org repo) was not following vcl rules set out to log the client IP address in an X-Forwarded-For header (see: Varnish 4 logging proxy/load balancer instead of client IP addresses) I happened to notice something odd while looking through varnishlog files:
- Begin req 9353447 rxreq - Timestamp Start: 1488771709.337974 0.000000 0.000000 - Timestamp Req: 1488771709.337974 0.000000 0.000000 - ReqStart 172.25.20.65 19903 - ReqMethod GET - ReqURL /about-us/ - ReqProtocol HTTP/1.1 - ReqHeader host: www.<notreallythishost>.com - ReqHeader Accept: */* - ReqHeader Accept-Encoding: gzip, deflate - ReqHeader Cache-Control: no-cache - ReqHeader From: bingbot(at)microsoft.com - ReqHeader Pragma: no-cache - ReqHeader User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) - ReqHeader X-Forwarded-For: 184.108.40.206 - ReqHeader X-Forwarded-Port: 80 - ReqHeader X-Forwarded-Proto: http - ReqHeader Connection: keep-alive - ReqUnset X-Forwarded-For: 220.127.116.11 - ReqHeader X-Forwarded-For: 18.104.22.168, 172.25.20.65 - VCL_call RECV - ReqUnset X-Forwarded-For: 22.214.171.124, 172.25.20.65 - ReqHeader X-Forwarded-For: 172.25.20.65 - ReqUnset Accept-Encoding: gzip, deflate - ReqHeader Accept-Encoding: gzip - ReqUnset User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 9353449 fetch - Timestamp Fetch: 1488771709.338395 0.000421 0.000421 - RespProtocol HTTP/1.1
This fully explains why we were never able to get anything other than the load balancer IP address logged through varnishncsa at any point regardless of the logging technique we used.
It looks as though in the request processing it builds up the X-Forwarded-For header by adding the AWS Load Balancer IP address to the header, but as vc_call is called summarily decontructs it again and removes the original client IP address. So how do I go about keeping the X-Forwarded-For intact and why is Varnish shifting IPs off the left side rather than than just adding them onto the X-Forwarded-For header like it should? Bug?