How can I best offload part of a domain's URIs from one server to another?

Question

Sorry, I am very new to this whole concept. Basically, I have a Django site. One of the view does a lot of work and leads to the server hanging up due to lack of memory. I want to take the functionality of that view and make it run on another server (Server B), and get the main server (Server A) to communicate with it like as if they both ran on the same server. My first guess Is I need to make Server B listen on a port that Server A can talk to and vice versa.

I'm not asking for a definite answer as this question in itself may be vague. Maybe, I can achieve this better by load balancing etc., I'm not sure. Any links pointing to information about how I can possibly achieve what I want is much appreciated. Thanks

Answer 1

I'd probably look towards some kind of messaging infrastructure to solve this. Server A can handle the front end requests, and server B can generate the data.

Server A "listens" on a queue for data generated by server B. You should also attempt to cache requests, and generated data so that you aren't regenerating the same data for multiple similar requests.

Server B listens for requests forwarded from the view on server A, processes the data and returns the generated data via the response queue.

Hmm. Things to read..

http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ <- Excellent article on RabbitMQ and python.

http://docs.djangoproject.com/en/1.2/topics/cache/ <- Django view caching. This will help lower your load immensely (especially if you use memcached over disk-based caching, then you get a distributed cache, too.)

http://oreilly.com/catalog/9780596102357 <- "Building scalable websites" will give you a good insight into some other ways to make your webapp scalable.

Answer 2

I'd suggest looking into common uses of Apache's mod_proxy.

Also, along this same path, you might get interested in something like Varnish for caching.

Think hard about ways to break up your application into functional units, if you haven't already, and to reduce the total amount of work done. Can you cache? If you can, do it. :)