Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I would like to have 2 identicals web servers : one master and one slave. File newly modified/create on the master should be replicated at once on the slave (in the minute).

I do not want to use rsync because it scans all the files to calculate the delta to send. I do not want to use a distributed file system like GLUSTER because I am afraid it can accept a lot of small write. Nevertheless I can accept to wait one minute to flush all the modifications to the slave.

Do you have an idea of what tool I should use ?

share|improve this question
    
I really think that your aversion to rsync is misplaced. –  gWaldo May 9 '11 at 12:40

8 Answers 8

I fail to see why you don't want to use rsync; that is, after all, exactly what it is for...

Since you say that you don't want to use a clustered filesystem, what about using the www folder on ServerA (share/export) to mount that on ServerB as the wwwRoot. Instead of replication, ServerB is using the exact same files.

share|improve this answer
    
by mounting the other servers www folder you put a dependency on the server you want to make a redundant copy of.. I don thtink that is the goal here. Tough rsync or drbd would work well for a www cluster in master/slave config. –  MrTimpi May 2 '11 at 12:38
    
Yup, agreed. But he doesn't want to use the standard tools that solve his problem... –  gWaldo May 2 '11 at 18:06

I have not tried it, but this may do what you are asking.

http://www.drbd.org/

share|improve this answer

If you keep your web application's files in version control, (you do have your files in version control, don't you?), you could write a script to pull those files down from your VCS and restart your web server service (Apache, NGINX, etc.). You could even have this run under cron so that every time you updated the repo (I recommend checking a Tag rather than a Branch or just Master) it automatically updates the website.

share|improve this answer

drbd would allow blocklevel replication, however, if you are doing any writes on the slave, you would either want to use OCFS2 or GFS to support cluster locking. If you can NFS mount the primary from the slave and could direct writes to the NFS mount, you could avoid using a cluster locking filesystem.

GlusterFS would be more seamless, but, lots of tiny writes do seem to get somewhat backlogged at times. OpenAFS is similar but almost any distributed filesystem would fit the bill. A two-node HDFS would probably also do what you need.

@gwaldo, As for not using rsync, if you have hundreds of thousands of files, it could take more than a minute just to traverse the tree to find the modified files.

share|improve this answer
    
@karmawhore, If you want someone to see and acknowledge your reply, you should do so as a comment on their post (like this). If you're going to build on what they say, then mention that, referencing the original. –  gWaldo Oct 3 '10 at 19:08

rsync shouldn't scan all files to calculate its deltas, by default it uses a quick-check algorithm that looks only for files with changed size or modifed-time. If you don't have many millions of files rsync run should be rather quick.

Otherwise you'll probably need a custom solution that will need to monitor the applications that can modify the data and send it over after the program closes the file.

share|improve this answer

DRDB + Heartbeat Apache cluster can be enabled ... personally i have tested this ... it is working in fine

share|improve this answer

You should probably use incrond to watch for changed files instead of running regularly (and blindly).

You would use the $# variable, which stands for the filename associated to the event, to sync each changed file individually. It may be the way to go if you want to avoid scanning all files in the path after each change. I haven't tried it though.

Also perhaps you should take a look at Unison:

Unison shares a number of features with tools such as configuration management packages (CVS, PRCS, Subversion, BitKeeper, etc.), distributed filesystems (Coda, etc.), uni-directional mirroring utilities (rsync, etc.), and other synchronizers (Intellisync, Reconcile, etc).

Here's a howto.

share|improve this answer

These requirements are not completely unusual, but it looks headed for trouble.

For most meaningful business web apps, high-available load-balancing proxies should be almost everywhere. This means using whatever is appropriate: dns round-robin, haproxy, ipvs, pfsense, f5's, netscalers, cisco ace, etc.

Web servers serving static content should be stateless. Other than dropping connections, there should be little to no impact on users if any one web server goes away. So, there is no need to have disk replication between machines. Use LB mentioned above to accomplish the same thing with less effort. Replication creates fragile dependencies and a support nightmare that could bring everything down. Pushing with git or rsync over ssh as mentioned previously from an internal deployment server is better idea. If pushing content to thousands of nodes, the murder gem from twitter is awesome.

Application servers, anything producing a web page based on data that changes, should be relatively stateless as well. Definitely use something like Nginx to do clean, rolling, exponential deployments of the app. Data should be kept in a database (sql/nosql) or RESTful data provider.

Exhaustively-tested failover should be reserved for protecting databases and other crucial components. For performance, if the application is measured to have a concurrent write bottleneck at the database beyond what throwing hardware at it (scale-up) can handle, consider a reliable nosql which uses a log-structured storage engine such as riak's bitcask.

If this is not a web app but processing a large amount of data, evaluate a MapReduce framework like Hadoop which uses their HDFS.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.