2

I have an architecture with two varnish servers sitting in front of 5 webheads. Each varnish server is configured with a round robin backend director, but at times of moderate to high load varnish seems to be heavily favouring the first defined backend in the list.

Varnish version is 3.0.5.

If the first backend is marked as sick the second backend in the list is heavily favoured, and so on.

varnish> backend.list
200
Backend name                   Refs   Admin      Probe
web1(************,,8080)       102    probe      Healthy 8/8
web2(************,,8080)       17     probe      Healthy 8/8
web3(************,,8080)       9      probe      Healthy 8/8
web4(************,,8080)       17     probe      Healthy 8/8
web5(************,,8080)       12     probe      Healthy 8/8

Some parts of the VCL that might be pertinent:

probe healthcheck {
   .request =
         "GET /LICENSE.txt HTTP/1.1"
         "Host: **********.co.uk"
         "Connection: close";
   .interval = 120s;
   .timeout = 90s; # High values due to expected slow responses
   .window = 8;
   .threshold = 3;
   .initial = 3;
   #.expected_response = 200; # Still want the Magento maintenance page to display so no response code check
}

backend web1 {
    .host = "************";
    .port = "8080";
    .connect_timeout = 240s; # High values due to expected slow responses
    .first_byte_timeout = 240s; # High values due to expected slow responses
    .between_bytes_timeout = 240s; # High values due to expected slow responses
    .probe = healthcheck;
}
backend web2 {
    .host = "************";
    .port = "8080";
    .connect_timeout = 240s; # High values due to expected slow responses
    .first_byte_timeout = 240s; # High values due to expected slow responses
    .between_bytes_timeout = 240s; # High values due to expected slow responses
    .probe = healthcheck;
}
backend web3 {
    .host = "************";
    .port = "8080";
    .connect_timeout = 240s; # High values due to expected slow responses
    .first_byte_timeout = 240s; # High values due to expected slow responses
    .between_bytes_timeout = 240s; # High values due to expected slow responses
    .probe = healthcheck;
}
backend web4 {
    .host = "************";
    .port = "8080";
    .connect_timeout = 240s; # High values due to expected slow responses
    .first_byte_timeout = 240s; # High values due to expected slow responses
    .between_bytes_timeout = 240s; # High values due to expected slow responses
    .probe = healthcheck;
}
backend web5 {
    .host = "************";
    .port = "8080";
    .connect_timeout = 240s; # High values due to expected slow responses
    .first_byte_timeout = 240s; # High values due to expected slow responses
    .between_bytes_timeout = 240s; # High values due to expected slow responses
    .probe = healthcheck;
}

director backend_director round-robin {
    { .backend = web1; }
    { .backend = web2; }
    { .backend = web3; }
    { .backend = web4; }
    { .backend = web5; }
}

sub vcl_recv {
    set req.backend = backend_director;

    # loads more stuff
}

Can anyone shed light on why the round robin director would so heavily favour the first defined backend, or what might cause a bypassing of the director entirely? I have already ensured that return(pipe) is not used in vcl_recv.

  • Do you mind sharing your healthcheck? How do response times from your backends compare? Are 'sticky sessions' in the picture? – KM. Apr 2 '14 at 16:32
  • @KM. Added the healthcheck. Not got data available for response times, but the imbalance in the backend list is reflected in the server load, which is significantly higher on whichever server varnish is favouring. – shanethehat Apr 2 '14 at 21:26
  • As a test, would it be possible to try using a random director with equal weight? "Equal weight means equal traffic" -- from: varnish-cache.org/docs/3.0/reference/… . How are you handling session cookies (if any)? – KM. Apr 3 '14 at 13:26
  • We were originally using a random director, and switched to round-robin in an attempt to fix the issue. It made no difference. Once a user is granted a session we start sending a nocache cookie that causes a return(pass). On the backend sessions are handled by a central memcached instance so sticky sessions are not required. – shanethehat Apr 3 '14 at 13:43
  • A theory has been floated that the excessively long timeout in the healthcheck might be the cause of the issue. – shanethehat Apr 4 '14 at 14:29

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.