4

We are trying to put varnish in front of our IIS 7.5 farm. Said farm handles 3 websites (SiteA, B and C) - varnish is going in front of A only.

When we add varnish, we get a 75-90% hit ratio but our IIS farm actually slows down so much that the requests which pass through to the IIS farm time out.

However, when we remove varnish, the entire SiteA runs 'fast enough' (~500ms response times) and our timeouts mostly vanish. Site B and C's performance are not affected.

Are there some settings in IIS that we should investigate? App pool timeout is set to the default of 20min, but since there's always traffic to these sites (30-200 req/sec to entire farm), I was under the impression the app pool timeout wouldnt be a factor.

Here's an average time_taken/min for SiteA in IIS during a normal time: time_taken (ms) - average per minute

Here's what it looks like when we go from IIS serving all traffic (green) and then plug in varnish (blue): hits/min vs time_taken/min

As you can see, ~40min after varnish starts serving most traffic, IIS (yellow) decides to slow down significantly. We've done this at various points in the day and it takes anywhere from 10min to 3hrs for us to witness our first massive slow down.

Recycling the app pools is what brings the speed back (blue jump up just before 15:30) but ultimately it slows down again rather quickly. Removing varnish and putting the full traffic against the IIS cluster (blue drop off) brings things back to speed.

Anyone have any ideas of what else to monitor and/or settings (IIS or varnish) to adjust?

Improve this question
2
  • Does this also happen when you don't allow varnish to cache anything, just proxy all requests?
    – gnur
    May 8, 2015 at 9:43
  • @gnur Thats one thing we're going to try next week - just pipe'ing data straight through varnish. However, we do see this same behavior when varnish isnt in the picture at all.. we just see it far, far less often (once a month or so).
    – Nate Fox
    May 8, 2015 at 17:21

1 Answer 1

Reset to default
1

It appears we found our issue - it had nothing to do with Varnish or IIS. The ASP.NET code had a bug in it. It was set up to create a connection to the DB and after X minutes of inactivity, it would close the connection. The bug was that it never tried to re-open a closed connection.

This makes sense if you took load off of the system, it started to collapse. It literally needed enough traffic to keep the connections open.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .