1

I'd love to harness the power of varnish to cache my php intensive application which serves around 400k people per day.

The application pulls in search data by firing off multiple threads which curl XML so you can imagine new threads spawn a lot and the threads stay open for a few seconds making page loads a couple of seconds.

A cache of each search result page would dramatically speed up the user experience.

So here is the foundation for my question.

Our search result pages require conversion code tracking. So user comes from source/referer A, to our page domain.com/search/?q=something&source=A, appropriate conversion tracking code (relating to referer A) is selected and outputted to the page. Cookie is also dropped so the next time the user returns the page checks if cookie exists, if yes, selects to show the correct conversion code in HTML.

This way the conversion tracking works in and out of session.

The question is, given the knowledge of our cookie requirements is it possible to use varnish in this scenario to cache? Can we somehow configure the VCL to handle these cookies and if so, what should we write?

Thank you

1
  • With those requirements the cache would need to vary based on the cookie (so for each distinct cookie, there will be a separate copy of any given results page in the cache), which puts a pretty big dent in the effectiveness of the cache. Can you change the page structure to where the search results don't need to vary, like if they were delivered via a JSON request separate from the variable aspects? Jun 2, 2013 at 23:08

1 Answer 1

5

I find that the easiest way to think about Varnish's effectiveness and implementation is by thinking in combinations. Each variable creates exponentially more combinations. In short, these variables are: host, URI, and headers/cookies.

e.g. these are different objects in Varnish cache

domain.com/search/?q=something
domain.com/search/?q=something&source=A
domain.com/search/?q=something&source=B
domain.com/search/?q=something&source=A + nocookie
domain.com/search/?q=something&source=A + cookie1
domain.com/search/?q=something&source=A + cookie2
domain.com/search/?q=something&source=B + nocookie
domain.com/search/?q=something&source=B + cookie1
domain.com/search/?q=something&source=B + cookie2

HOWEVER: So long as the sources don't vary much, and so long as the server isn't responsible for outputting different content based on the source, it should be semi-straightforward to use Varnish... but only if you do some manipulation first.

Since you have the ability to manipulate much of the client's request with Varnish, its possible for you to actually strip the &source=A or &source=B from the requested URI before it's sent to the backend server. This essentially turns all of these requests:

domain.com/search/?q=something&source=A + nocookie
domain.com/search/?q=something&source=A + cookie1
domain.com/search/?q=something&source=A + cookie2
domain.com/search/?q=something&source=B + nocookie
domain.com/search/?q=something&source=B + cookie1
domain.com/search/?q=something&source=B + cookie2

into just this:

domain.com/search/?q=something

What was 6 misses and no hits is now 1 miss and 5 hits

So client requests this from Varnish:

domain.com/search/?q=something&source=A + cookie1

and Varnish actually requests this from the backend (e.g. Apache) for the first request:

domain.com/search/?q=something

and then is cached for subsequent requests (thereby increasing your hit rate significantly). This is called "normalizing".

Then of course the static JavaScript file will do its work by referencing the URI query strings and do some DOM manipulation (kind of what Google Analytics does) based on the source query string.

So for the client, the &source=A will be maintained and JavaScript can use that accordingly; and so long as the JavaScript is responsible for dynamically changing the content, you should have no problem stripping them out all or most of the cookies or query strings from your request before Varnish sends the request to the backend.


You can also cache your XML requests so long as they're GET requests.

Basically the name-of-the-game with Varnish is all about "normalizing" the backend request so that URIs/cookies/headers that don't effect what's returned from the server should be manipulated aka normalized before being sent to the backend

Reformatting the URI in Varnish: https://stackoverflow.com/questions/3547384/can-i-reformat-my-url-parameters-with-varnish

Should you have a need to cache content dynamically based on a cookie, you can do so using vcl_hash: https://www.varnish-cache.org/trac/wiki/VCLExampleCacheCookies This of course lowers your hit-rate so its far better to pass off such functionality to JavaScript to handle and tell Varnish not to cache specific endpoints: e.g.

// don't cache this endpoint, this content changes based on the referrer
if (req.url ~ '/ajax/get_referrer/') { return (pass); }

The only part I don't understand in your question was:

Cookie is also dropped so the next time the user returns the page checks if cookie exists, if yes, selects to show the correct conversion code in HTML.

So long as the backend server doesn't need to see the cookie or set the cookie, that is, so long as JavaScript is responsible for taking care of the DOM work, you should be in the clear. Note that if the 'source/referrer' is different per-user, you should also tell Varnish not to cache any endpoints used to get the data you need.

You should also note that you should only cache GET and HEAD requests in Varnish. If your search or JavaScript uses POST or any other request type, they should not be cached.


I definitely recommend doing everything on a development server. You'll have many other factors to consider such as delivering PDFs/videos/audio (aka piping requests), ignoring pages, and many more considerations unique to your situation.

1
  • +1. Great first answer!
    – EEAA
    Jul 3, 2013 at 22:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .