Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have servers spread across several data centers, each storing different files. I want users to be able to access the files on all servers through a single domain and have the individual servers return the files directly to the users.

The following shows a simple example: 1) The user's browser requests http://www.example.com/files/file1.zip 2) Request goes to server A, based on the DNS A record for example.com. 3) Server A analyzes the request and works out that /files/file1.zip is stored on server B. 4) Server A forwards the request to server B. 5) Server B returns file1.zip directly to the user without going through server A.

Note: steps 4 and 5 must be transparent to the user and cannot involve sending a redirect to the user as that would violate the requirement of a single domain.

From my research, what I want to achieve is called "Direct Server Return" and it is a common setup for load balancing. It is also sometimes called a half reverse proxy.

For step 4, it sounds like I need to do MAC Address Translation and then pass the request back onto the network and for servers outside the network of server A tunneling will be required.

For step 5, I simply need to configure server B, as per the real servers in a load balancing setup. Namely, server B should have server A's IP address on the loopback interface and it should not answer any ARP requests for that IP address.

My problem is how to actually achieve step 4?

I have found plenty of hardware and software that can do this for simple load balancing at layer 4, but these solutions fall short and cannot handle the kind of custom routing I require. It seems like I will need to roll my own solution.

Ideally, I would like to do the routing / forwarding at the web server level, i.e. in PHP or C# / ASP.net. However, I am open to doing it at a lower level such as Apache or IIS, or at an even lower level, i.e. a custom proxy service in front of everything.

Thanks.

share|improve this question

9 Answers 9

Daniel - you could probably get this done with LVS and a clustered filesystem relatively easily. Your LVS load balancers would receive the HTTP requests and they would forward the requests to a web node that can respond to the request. The response would leave directly from the web node and the load balancer wouldn't be involved at all in the reply.

I put together a guide for this (on Fedora) that works well on virtualized Xen instances.

share|improve this answer

I think, you should use either caching proxy server or HTTP redirects to achieve your goal.

For example, with nginx you can use a backend (PHP script or some FastCGI-hosted server) to determine which 'datacenter' you should go, return X-Accel-Redirect to new location with proxy_pass to that datacenter. After this, nginx should cache/store your result.

So, you config in every datacenter will be different only for files that are 100% local.

The second case - backend returns 301 error code with necessary direct link.

share|improve this answer
    
It sounds like you're suggesting the Anycast solution along with a clever trick to achieve it. Whilst this would work well for ensuring the files are replicated, it wouldn't work too well for the site itself as it is heavily database driven and general caching is not suitable for that. Without caching, the additional overhead of routing simple web page requests through additional servers (over great distances) would slow things down too much. HTTP redirects would be ideal. However, due to security provisions in Flash, they won't work. See my comments to Jesper for more details. –  Daniel Crabtree Feb 15 '10 at 1:14
    
As I see, you'll need access to BGP level of networking in every datacenter. So you can advertise route to your several servers throw number of routes. Then you can set up something like request-transferring mechanism to transfer request received by one server and the client connection to another datacenter through some trusted connection (secure VPN). After all, the second datacenter can send response directly to client like in DSR. So, inside VPN this will look like clear DSR, but the VPN itself will have several exit gateways - one for each datacenter. –  Yorik.sar Feb 21 '10 at 19:42

Daniel, you have really done your homework. Maybe there is a Guru out there to prove me wrong, but my initial response is "don't do that". I know that's sort of a non-answer, but without any reasoning as to why you truly need this, it's the best answer that I have. While I think the proposed solution could work, it's a lot of configuration and setup to handle just to hide a domain -- it seems too brittle and complex to me.

I would suggest either:

  • Play negotiation hardball, and negotiate that single domain requirement away. Really explain how much this increases complexity and cost. Then use plain HTTP redirects to each server / site.
  • OR
  • Replicate all data to all data centers (DCs) for redundancy. Use Anycast to make one domain available on multiple DCs. Use a conventional load balancer of your choice at each DC.

HTH

share|improve this answer
    
It would be great if we could just use plain HTTP redirects and this would be preferred. However, the files being distributed are Flash SWF files and most use local storage, which requires the domain/subdomain stay constant. For redundancy, sometimes a user for file A might be directed to server B and other times to server C, if they have different names, that breaks local storage. –  Daniel Crabtree Feb 15 '10 at 1:03
    
I have considered replicating the data everywhere using a single separate IP and subdomain, to distinguish the distributed content from the website. This would allow the layer 4 load balancing solution to work (for which there are hardware solutions). However, users can disable 3rd party storage in Flash and I think this might even be the default setting for everyone now. This means I don't think it is even an option to move it all to a single sub-domain and that it needs to stay on the single domain. –  Daniel Crabtree Feb 15 '10 at 1:04
    
Anycast would probably work. However, replicating the database to every location and keeping it syncronished is a problem we don't need at this stage. We mostly want to move the large files closer to users to distribute the load, to speed up those downloads, and to provide some redundancy for them. The main site would at most want to be at 2 or 3 sites, whereas already we would like about 4 or 5 locations for file servers. –  Daniel Crabtree Feb 15 '10 at 1:06
    
@Daniel: Good clarifications. I don't have a better answer for you -- let the question be open here, and see what comes up, or hire a global load balancing expert. :-) Let us know how it goes. –  Jesper Mortensen Feb 15 '10 at 9:09
    
Thanks for your help. I've contacted a few of the companies that make load balancing hardware to see if they have any solutions, maybe one of them will have the answer and hopefully it isn't too expensive. –  Daniel Crabtree Feb 15 '10 at 12:32

Any commercial load balancer can be configured for Direct Server Return. (F5 BIG-IP can do it; in their terminology, it's called "nPath routing".) A low-cost solution is Linux Virtual Server. The Direct Routing setup is documented here.

However, the catch is that for the load-balancing to be transparent to Layer 3, the forwarding has to be done at Layer 2. That means that your load balancer and the content servers have to be on the same physical segment. I'm not sure how you would accomplish that over several datacenters.

Alternatively, you HTTP-redirect the page containing the Flash application. Then you would have independent sites, each satisfying the same-origin principle.

share|improve this answer

A combination of Linux Virtual Server and keepalived and maybe some arptable magic should accomplish what you're seeking. I've used that very method extensively in the past and it's proven to be very scalable. The fact that the real servers talk back to the client after receiving the connections makes the load balancer (LVS+Keepalived) very scalable.

The trick is to get the arptables correct. On Red Hat/CentOS, the package is called arptables_jf. Here are some sample rules to place on Server B from your example, assuming 10.10.0.1 is you VIP and 10.10.0.10 is Server B's IP:

arptables -A IN -d 10.10.0.1 -j DROP 
arptables -A OUT -s 10.10.0.1 -j mangle --mangle-ip-s 10.10.0.10

Note that your load balancer (Server A) and real server(s) (Server B+) should sit on the same network.

You also need to configure each real server with a "dummy" interface with the IP of the VIP. Again, on Red Hat/CentOS:

# cat /etc/sysconfig/network-scripts/ifcfg-dummy0
DEVICE=dummy0
BOOTPROTO=static
IPADDR=10.10.0.1
NETMASK=255.255.255.255
ONBOOT=yes
TYPE=Ethernet

The arptable rules will handle silencing ARP requests for that IP (without arptables, all real servers and load balancer will answer the question: who has 10.10.0.1).

I hope this helps!

share|improve this answer

Doing DSR via arp mangling and some other trickery works at the local level - if you want to extned this out to a bunch of, let's call them "slave" data centers - you will need to have the packets encapsulated and send to another box at the remote DC to remove the encapsulation - could be your nodes individually or it could be another instance of a load balancer doing regular arp-based DSR.

If you are doing this across multiple DCs, you are also going to need to ensure their filtering will allow the outgoing traffic from IP addresses that they don't manage, and that any firewalling devices in the way that may be flow-based are not going to interfere - as they are only going to see one side of the conversation (the other side comes in encapsulated)

Just an addition - can you provide some insight into why you need this particular setup? What exactly are you trying to achieve by doing this rather than using redirects or multiple domain names. If multiple domain names were okay for the actual resource files - a CDN or an Anycast solution would be a far more economical and reliable solution - let your app logic determine which resource to redirect users to.

What you are proposing is technically interesting, but it's also creating a single point of failure - if you are having difficulty figuring this out on a local, layer-2 level with arp spoofing, then handling it at a global level is only going to be more complicated.... a CDN is probably what you need.

That said - from a purely network point of view,this is really interesting - if you can provide a bit more detail, I can probably work out a specific solution using Linux tools that will do what you are asking - sounds fun.

share|improve this answer

I you can't drop the "same domain" requirement and don't want to go to the advanced networking route, maybe you can drop the "storing different files at different locations" requirement?

If you can get all servers (locations) to store all files, you could have identically configured stacks behind a round-robin IP behind a single domain. Every location would be fully capable and reachable behind a single domain. Any database action would still carry a replication or remote connection requirement, of course.

I make the distinction between servers and locations, because reverse proxying over a high speed low latency LAN is not that bad (see Yorik's answer).

share|improve this answer

How about application request routing?

http://www.iis.net/download/ApplicationRequestRouting

Your backend storage servers can use DFS to keep your content in synch at the back and it will ensure that the server accessing the content will always take the shortest route to the content.

share|improve this answer

Daniel,

The workaround for this should be in the application layer. If you do it on the lower layers, the rules will become applicable to all traffic, while here, it is application to a specific application.

I would suggest that you create subdomains and give the IP addresses of the different servers as an A record on the subdomains. THerefore, since it is in the same domain, the third party deal gets taken care of. Then use a database in the central server and make CURL requests to the specific servers. That way, your domain is also hidden, while your application works. However, the only problem here is that the database needs to be speckless at all times. Otherwise you application will fail miserably.

See to it that the database runs at all times and efficiently. You should probably consider making a backup data bank so that you don't have to go through the pitfalls of server downtime.

Regards
Binaek Sarkar
Foundation

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.