I currently have a bunch of NameVirtualHost based websites, load balanced between 3 apache2 servers using ldirectord. I would like to insert varnish as a reverse-web-proxy between ldirectord and apache in the following way:

  1. a request comes in to ldirectord
  2. it is then load balanced between the 3 apache2 servers and varnish, with a weight of 1 for the webservers, and 99 for varnish (so if varnish is rebooted, the webservers will take over seamlessly)
  3. varnish will then load balance its requests between my apache2 servers.

However, the varnish part is not working.

I wonder whether this has to do with the fact that my apache servers use x.x.x.x:80 for their NameVirtualHosts, instead of *:80? (they have to do this, since each server hosts multiple IP addresses)

Or perhaps it has to do with the need for IP Forwarding to be set up on the varnish server? (I did echo 1 > /proc/sys/net/ipv4/ip_forward on this server, is that sufficient?)

How can I debug this problem?

  • ldirectord doesn't produce logs of what it does with each request (and if it did, I would be overwhelmed with information since I'm serving hundreds of requests per second)
  • varnish log shows the ldirectord server connecting to it every 5 seconds, but nothing else.
  • I have set up a test site using this configuration, but it fails - no apache access logs, no applicable varnish logs.

You don't need to enable IP Forwarding on the varnish server for it to work. Varnish will not forward the client connection but, as a proxy, create a new connection on the user's behalf.

If you can

telnet apache2 80


curl -x apache2:80 http://yoursite.com/yourpage

from the varnish server then your network setup is ok. For the latter example please add the code below to your vcl_recv config:

# Normalise requests sent via curl's -X mode and LWP.  Must do before
# backend selection.
if (req.url ~ "^http://") {
  set req.url = regsub(req.url, "http://[^/]*", "");

It would help if you had posted your varnish configuration but the original one should work out of the box (even if it's not caching a great deal of pages).

I assume you created a director for your 3 apache backends and that this director is the default backend for all incoming connections to varnish.

If so, run

varnishlog | grep _health

and make sure your backends are not sick. If so, adjust your backend probe (health check).

As a rule of thumb, varnish does not care what virtualhost configuration is being used in the backends. I suggest you go back to the original configuration and define a single backend to start with. Then move on to a director. Only then further customise your varnish config.

good luck

  • Thanks for the info. Unfortunately I cannot address the individual webservers by the same IP address that apache is listening on, because that IP address is a virtual IP shared between 3 servers (the config files are also shared), and load balanced by ldirectord. This will create a loop. And of course, addressing the servers by their REAL IP address does not work because apache is not listening on that address. Make sense? – Brent Jun 9 '10 at 21:57
  • I think so - I'm not too familiar with LVS, my apologies. Short of making varnish your public server I don't see an easy way of doing this. Unless you want to run an instance of varnish on each apache2 server but that's hardly optimal. What if HA 2 Varnish servers and make them the entry-point instead? I've only used Varnish behind a load-balancer appliance so I never had to worry about making a varnish instance highly-available :-/ – Thiago Figueiro Jun 9 '10 at 22:06

I assume you use LVS/DR (or LVS/TUN) because of the shared ip-address. I'm no network expert, but as I understand it, ldirectord can send packets to a specific back-end with a shared ip-address by setting the right mac-address in the ethernet frame, and leave the ip-address of the TCP/IP packet unchanged. Varnish can not do this (it works on TCP/IP-level).

One solution is to make your apache2 services (also) listen to a ip-address/port combination which varnish can address and your network can route. Can you add an internal-network ip-address to each of the apache2 servers and listen to this address as well?

Another solution could be to use fake addresses at the varnish server to address each of the apache servers, and have an iptables outgoing firewall chain mangle the packets correctly (set the shared ip-address, set a specific mac-address).

The simplest thing would be to use MASQ (LVS/NAT) for your apache servers. Both LVS and varnish can address them at the same ip-address (LAN or public).

Good luck!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.