Forwarding HTTP Request with Direct Server Return

Question

I have servers spread across several data centers, each storing different files. I want users to be able to access the files on all servers through a single domain and have the individual servers return the files directly to the users.

The following shows a simple example: 1) The user's browser requests http://www.example.com/files/file1.zip 2) Request goes to server A, based on the DNS A record for example.com. 3) Server A analyzes the request and works out that /files/file1.zip is stored on server B. 4) Server A forwards the request to server B. 5) Server B returns file1.zip directly to the user without going through server A.

Note: steps 4 and 5 must be transparent to the user and cannot involve sending a redirect to the user as that would violate the requirement of a single domain.

From my research, what I want to achieve is called "Direct Server Return" and it is a common setup for load balancing. It is also sometimes called a half reverse proxy.

For step 4, it sounds like I need to do MAC Address Translation and then pass the request back onto the network and for servers outside the network of server A tunneling will be required.

For step 5, I simply need to configure server B, as per the real servers in a load balancing setup. Namely, server B should have server A's IP address on the loopback interface and it should not answer any ARP requests for that IP address.

My problem is how to actually achieve step 4?

I have found plenty of hardware and software that can do this for simple load balancing at layer 4, but these solutions fall short and cannot handle the kind of custom routing I require. It seems like I will need to roll my own solution.

Ideally, I would like to do the routing / forwarding at the web server level, i.e. in PHP or C# / ASP.net. However, I am open to doing it at a lower level such as Apache or IIS, or at an even lower level, i.e. a custom proxy service in front of everything.

Thanks.

Answer 1

Daniel - you could probably get this done with LVS and a clustered filesystem relatively easily. Your LVS load balancers would receive the HTTP requests and they would forward the requests to a web node that can respond to the request. The response would leave directly from the web node and the load balancer wouldn't be involved at all in the reply.

I put together a guide for this (on Fedora) that works well on virtualized Xen instances.

Answer 2

I think, you should use either caching proxy server or HTTP redirects to achieve your goal.

For example, with nginx you can use a backend (PHP script or some FastCGI-hosted server) to determine which 'datacenter' you should go, return X-Accel-Redirect to new location with proxy_pass to that datacenter. After this, nginx should cache/store your result.

So, you config in every datacenter will be different only for files that are 100% local.

The second case - backend returns 301 error code with necessary direct link.

Answer 3

Daniel, you have really done your homework. Maybe there is a Guru out there to prove me wrong, but my initial response is "don't do that". I know that's sort of a non-answer, but without any reasoning as to why you truly need this, it's the best answer that I have. While I think the proposed solution could work, it's a lot of configuration and setup to handle just to hide a domain -- it seems too brittle and complex to me.

I would suggest either:

• Play negotiation hardball, and negotiate that single domain requirement away. Really explain how much this increases complexity and cost. Then use plain HTTP redirects to each server / site.
• OR
• Replicate all data to all data centers (DCs) for redundancy. Use Anycast to make one domain available on multiple DCs. Use a conventional load balancer of your choice at each DC.

HTH

Answer 4

Any commercial load balancer can be configured for Direct Server Return. (F5 BIG-IP can do it; in their terminology, it's called "nPath routing".) A low-cost solution is Linux Virtual Server. The Direct Routing setup is documented here.

However, the catch is that for the load-balancing to be transparent to Layer 3, the forwarding has to be done at Layer 2. That means that your load balancer and the content servers have to be on the same physical segment. I'm not sure how you would accomplish that over several datacenters.

Alternatively, you HTTP-redirect the page containing the Flash application. Then you would have independent sites, each satisfying the same-origin principle.

Answer 5

A combination of Linux Virtual Server and keepalived and maybe some arptable magic should accomplish what you're seeking. I've used that very method extensively in the past and it's proven to be very scalable. The fact that the real servers talk back to the client after receiving the connections makes the load balancer (LVS+Keepalived) very scalable.

The trick is to get the arptables correct. On Red Hat/CentOS, the package is called arptables_jf. Here are some sample rules to place on Server B from your example, assuming 10.10.0.1 is you VIP and 10.10.0.10 is Server B's IP:

arptables -A IN -d 10.10.0.1 -j DROP 
arptables -A OUT -s 10.10.0.1 -j mangle --mangle-ip-s 10.10.0.10

Note that your load balancer (Server A) and real server(s) (Server B+) should sit on the same network.

You also need to configure each real server with a "dummy" interface with the IP of the VIP. Again, on Red Hat/CentOS:

# cat /etc/sysconfig/network-scripts/ifcfg-dummy0
DEVICE=dummy0
BOOTPROTO=static
IPADDR=10.10.0.1
NETMASK=255.255.255.255
ONBOOT=yes
TYPE=Ethernet

The arptable rules will handle silencing ARP requests for that IP (without arptables, all real servers and load balancer will answer the question: who has 10.10.0.1).

I hope this helps!

Answer 6

Doing DSR via arp mangling and some other trickery works at the local level - if you want to extned this out to a bunch of, let's call them "slave" data centers - you will need to have the packets encapsulated and send to another box at the remote DC to remove the encapsulation - could be your nodes individually or it could be another instance of a load balancer doing regular arp-based DSR.

If you are doing this across multiple DCs, you are also going to need to ensure their filtering will allow the outgoing traffic from IP addresses that they don't manage, and that any firewalling devices in the way that may be flow-based are not going to interfere - as they are only going to see one side of the conversation (the other side comes in encapsulated)

Just an addition - can you provide some insight into why you need this particular setup? What exactly are you trying to achieve by doing this rather than using redirects or multiple domain names. If multiple domain names were okay for the actual resource files - a CDN or an Anycast solution would be a far more economical and reliable solution - let your app logic determine which resource to redirect users to.

What you are proposing is technically interesting, but it's also creating a single point of failure - if you are having difficulty figuring this out on a local, layer-2 level with arp spoofing, then handling it at a global level is only going to be more complicated.... a CDN is probably what you need.

That said - from a purely network point of view,this is really interesting - if you can provide a bit more detail, I can probably work out a specific solution using Linux tools that will do what you are asking - sounds fun.

Answer 7

I you can't drop the "same domain" requirement and don't want to go to the advanced networking route, maybe you can drop the "storing different files at different locations" requirement?

If you can get all servers (locations) to store all files, you could have identically configured stacks behind a round-robin IP behind a single domain. Every location would be fully capable and reachable behind a single domain. Any database action would still carry a replication or remote connection requirement, of course.

I make the distinction between servers and locations, because reverse proxying over a high speed low latency LAN is not that bad (see Yorik's answer).

Answer 8

How about application request routing?

http://www.iis.net/download/ApplicationRequestRouting

Your backend storage servers can use DFS to keep your content in synch at the back and it will ensure that the server accessing the content will always take the shortest route to the content.

Answer 9

Daniel,

The workaround for this should be in the application layer. If you do it on the lower layers, the rules will become applicable to all traffic, while here, it is application to a specific application.

I would suggest that you create subdomains and give the IP addresses of the different servers as an A record on the subdomains. THerefore, since it is in the same domain, the third party deal gets taken care of. Then use a database in the central server and make CURL requests to the specific servers. That way, your domain is also hidden, while your application works. However, the only problem here is that the database needs to be speckless at all times. Otherwise you application will fail miserably.

See to it that the database runs at all times and efficiently. You should probably consider making a backup data bank so that you don't have to go through the pitfalls of server downtime.

Regards
Binaek Sarkar
Foundation