3

I have a file server, which is in-charge of pulling a folder tree from multiple workstations on a daily basis. My current method for this is by using rsync, (which works pretty well provided directory names and/or files remain the same) however, when files are renamed or moved about within subdir1, rsync will copy them over to the server, creating duplicates.

I have to manually find and delete extraneous files/folders that had been left on the server during previous syncs. Note that I cannot use rsync's --delete flag because any sync from a workstation will then mirror that particular folder tree, instead of merging them to the server.

Visual diagram:

Server:         Workstation1        Workstation2        Workstation(n)
Folder*         Folder*             Folder*             Folder*
-subdir1        -subdir1            -subdir1            -subdir(n)
  -file1          -file1              -file2              -file(n)
  -file2
  -file(n)
  • Is there a simple script (preferably in bash, nothing fancy) that can accomplish the deletion of the extraneous files/folders in the event a file is renamed or moved to a different subdir?
  • Is there a different program, much like rsync that can accomplish this task autonomously and in a much simpler manner? I have looked at unison, but I did not like the fact that it keeps a local database for the syncing info.
  • Any tips at all as to how I am supposed to tackle this?

Thank you in advanced for your help.

EDIT: I have tried unison just recently and I can safely say it is out of the question now. unison is a bi-directional synchronization tool and from my testing, it mirrors the files existing on the server to all workstations. - This is unwanted.

preferably, i would want files/folders to stay within their respective workstations and just merge to the server. AKA uni-directional sync; but with renames/moves propagated to the server.

I might have to look into Git/Mercurial/Bazaar as mentioned by kyle, but still unsure if they are fit for the job.

5
  • If I were to use Git/Mercurial/Bazaar for this, can it be done so that the commit and whatnot be done via scripts/automated? - forgive the question, I have no experience in using any of those utilities. Mar 22, 2012 at 18:59
  • Could you clarify what you mean by "Note that I cannot use rsync's --delete flag because any sync from a workstation will then mirror that particular folder tree, instead of merging them to the server"?
    – amcnabb
    Mar 22, 2012 at 19:04
  • @amcnabb if you look at the diagram, files from workstation1,2,etc are synced to the server. Now, if I were to use rsync --delete workstation1/Folder/ server:/Folder the process would remove file2 from workstation2, and file(n) from workstation(n), etc... Leaving only workstation1's file. Mar 22, 2012 at 19:11
  • Then how can you tell the difference between Workstation1 moving the file and Workstation1 never having the file to begin with? If you need such clever logic, then Unison is a good solution.
    – amcnabb
    Mar 22, 2012 at 19:42
  • If workstation1 did not have the file to begin with, it would not exist on the server. File1 is there because workstation1 had it before, then let's say he/she moved it to subdirX, or even renamed subdir1 to subdirX, the server would have copies of both /subdir1/file1 and /subdirX/file1 Mar 22, 2012 at 19:52

1 Answer 1

4

I think Unison or even Git/Mercurial/Bazaar may be a better choice for this.

A bash script would be extremely difficult to do since the rename of a file would be impossible to differentiate from the creation of a new file, which is what you're running into with rsync.

If it were me I would try to use Git or Mercurial. That would probably require some additional user training, but it would really be the best option. Also, it gives you the benefit of version control so you can revert mistakes.

6
  • I have long wanted to try other solutions as you mentioned, unison/git or other subversion utils, but the sheer complexity they have compared to rsync is what's stopping me. Also, if I were to use git, will the merged folder on the server be reflected on the individual workstations too? -cause this cannot happen. Files belonging to the workstations must stay in their respective workstations. Mar 22, 2012 at 18:29
  • +1 for Mercurial in this situation. It's simple enough that your users should be able to use it with a small amount of training and way more appropriate for the task at hand. Mar 22, 2012 at 18:29
  • @ReyLeonardAmorato Git is much more complex than Mercurial, but also really good for customizing. Mercurial isn't too bad as far as complexity so if you can do maybe an hour training session with users you will probably be ok. If you did do git/hg it would be hard if not impossible to merge the files properly without the changes being reflected on the other workstation's machine. Is there a particular reason the workstation can't be allowed to see other work station's changes?
    – Kyle
    Mar 22, 2012 at 20:05
  • Hm. So, what you are saying is that the users have to go to the terminal and manually commit their changes to the repository? Can't it be done automatically? Say after office hours or so.. These people have no interest whatsoever in the technical stuff. Mar 22, 2012 at 20:10
  • 1
    Maybe take a look at this: github.com/ianb/git-sync it allows you to use Git basically as rsync, but it should keep track of all file changes. Then just hook it up as a cronjob or scheduled task. The only thing you may have to worry about is file conflicts on the server.
    – Kyle
    Mar 22, 2012 at 20:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .