Nginx/haproxy/IIS - Configuration questions and concerns

Question

Our environment is mostly consisting of SSL web sites running on IIS/ASP.Net. The majority of the site code is reporting, analysis, and user specific views of data. We are expanding the number of concurrent users and I need to plan for failover and cluster capability. I've been doing research on setting up nginx and haproxy as well as possibly varnish to handle the SSL offloading, load balancing, and caching.

One of my questions pertains to the configuration. I believe, from what I have read, nginx needs to be in front of varnish due to the SSL sites we have running. I envision the proper setup to be:

*-------*
* nginx *
*-------*
    |
*---------*
* varnish *
*---------*
    |
*---------*
* haproxy *
*---------*
    |\
    | - - - - -
    |         |
*------*   *------*
* web1 *   * web2 *
*------*   *------*

First, is this the best case setup for our environment (IIS 7.5) and is there any benefit to implementing varnish in a case where most of the content is essentially per request dynamic content? I'm hitting the point of paralysis by analysis on this topic and can't seem to get to a good answer on the use cases for a varnish middle layer.

My second question is dealing with single point of failure. Right now our ASA5510 routes to each web server so our single point is the firewall/connection. The connection has such a small chance of even going down that I am really only considering the firewall as the single point of failure in our configuration. Management is sure to look at the above diagram and see 3 possible new points of failure. Is there a way to setup this up to eliminate any of the points as main bottlenecks to all servers? I couldn't find consistent reports of instability with nginx or haproxy so they seem to be solid, but I would still like to provide failover wherever possible.

Any thoughts or ideas as to the best way to get these integrated without compromising the infrastructure?

Answer 1

Personally I'd start by just using nginx - varnish is faster when your bottleneck is static (or at least semi-static) content, but from your description it seems that nginx's built-in caching features would be good enough; similarly with haproxy, it is a better load balancer than nginx, but unless you're pushing the limits of load balancing, nginx can probably take care of the simple stuff quite happily.

That said, if you are pushing the limits of caching and load balancing, I've found that having all three at once is fine, the only minor annoyance being keeping track of the client IP address through each layer.

Answer 2

That's a similar setup to ours, except we don't have varnish in the middle. The nginx and HAProxy "servers" are on the same box. nginx does SSL offload for us, and then HAProxy sends the requests to one of several web servers. We wanted the enhanced load balancing from HAProxy so we have both. The entire process works great.

The only gotcha we found was the client ip address. We couldn't find a way to tell IIS to log the x-forwarded-for IP as the clientip in its logs, but we did find that ASP.Net can ask IIS to append some additional information to the end of the request string in the log, so we stick the client IP there. Not ideal, but it works. nginx is set to add a x-forwarded-for header, HAProxy is not set to add forwarded for information (since it would simply point to nginx). Also, nginx seems to only add the header for the first request in a keep-alive session -- again, not ideal, but it's good enough for us.

Excerpt of our nginx config (HAProxy is listening on 127.0.0.1:5000):

location / {
    proxy_pass      http://127.0.0.1:5000;
    proxy_set_header        Host    $host;
    proxy_set_header        X-Forwarded-For $remote_addr;
    proxy_buffering off;
    client_body_buffer_size 64k;
}

ASP.Net code (in global.asax) to add the ip to the request header:

protected void Application_BeginRequest(Object sender, EventArgs e)
{
    //If there's an X-ForwardedFor header (Proxy) append it to the query string for logging purposes.
    if (HttpContext.Current.Request.Headers["X-Forwarded-For"] != null)
        HttpContext.Current.Response.AppendToLog((HttpContext.Current.Request.QueryString.Count == 0 ? "" : "&") + "X-Forwarded-For=" + HttpContext.Current.Request.Headers["X-Forwarded-For"]);