5

I have an app farm which generates thumbnails and makes them available to a web server farm using nfs. However the performance is so poor so I am going to make a local copy of the files on each web node.

I looked at unison and lsync, but there doesn't seem to be a rpm available from the centOS5.5 repo for either.

which one should I use, or some other strategy. I critically need the sync to be immediate as the user needs to display the thumbnail straight away on a web page, so inotify based tools are good!

Edit: when I say immediate, I mean pushed out rather than pulled on some schedule.

In the end, lsync have released some more versions so I used that, and lsync works well once it is running (ie syncs in a few seconds) however it needs to enumerate each directory beforehand and for a very large file-system takes many hours..

Improve this question

3 Answers 3

Reset to default
1

I critically need the sync to be immediate as the user needs to display the thumbnail straight away on a web page

So you've no control over server affinity in your load balancing? Time to get a new load balancer.

I looked at unison and lsync, but there doesn't seem to be a rpm available from the centOS5.5 repo for either.

Previously I did this by building replication into the application - after all, it's the first thing to know when new content needs to be replicated - it was just a few lines of PHP to publish a notice to the other nodes that content needed fetching - when a node has been offline for a while, I ran rsync before bringing it completely back online (this approach makes handling offline nodes much simpler and non-intrusive). However if I were to implement such a system today then I do it using overlay filesystems - perhaps an NFS at the bottom of the stack and copy up on access.

See also this post

HTH

C.

Improve this answer
1
  • well the idea thing would be to build something into the app, and preferably better design the nfs layout. but thats not possible at the moment
    – Tom
    Jul 9, 2010 at 3:20
0

I don't see anything for it in the repository, but you might find that incron may be helpful.

Improve this answer
1
  • hi, thanks for the reply. incron is not recursive. though it looks kinda cool and i will be messing with it for fun at some later date.
    – Tom
    Jul 9, 2010 at 3:19
0

I recently installed unison version 2.32.52 on a one of my CentOS 5.x boxes.

With unison its important to note that all servers that will be synchronizing together will need to be running the exact same version. Unison won't work when linked to other unison versions older and newer.

For myself I found that this rpm http://rpmfind.net/linux/RPM/dag/redhat/el5/x86_64/unison-2.32.52-1.el5.rf.x86_64.html worked for me running on centos 5.8

As for lsync I wouldn't be able to tell you much about it as I have never personally used it. I can however say that Unison when used properly is a fantastic tool.

You can set Unison on a crontab to check for changes every 1-10 minutes.

Improve this answer
1
  • Unison now has a repeat=watch option to monitor file changes.
    – RJ-
    Sep 23, 2015 at 2:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .