0

We have an online production server (source, half-history) which contains recent N days (or M months) audio files, we want to backup/archive/synchronize those audio files to backup server(s) (destination(s), full-history). When the destination disk is about to be full, we will power off the backup server, and install a new hard drive disk, but we do not want backup/archive/synchronize the files which are already backuped again.

We want to use open source tool, say rsync, to do this job. But once if the new hard drive disk is installed, rsync will re-synchronize from the source.

  • Is there a way to let rsync remember synchronized files and do not synchronize them again ?
  • or, is there another open source solution can meet the requirement ?

--Edit--

I forgot to mentioned that

  • The source server is running Windows Server 2003 system with Cygwin installed, and the backup servers can be intalled linux system, but better use windows client system, because the users are not technicians.
  • The total file size and file amount are large, it tooks me 80+ minutes to get the du information of files and another 80+ minutes to get file list of files which were produced in a single month 2012-07. find -type d -name 201207* -exec du -s {} \; > 2012-07-dir-stats.txt: 382 directories, 267 GB (279715008 KB). find -type f -name 201207* > 2012-07-file-list.txt: 1093962 files (file size of list file itself is 80+ MB.
  • The source server contains recent N days audio files means: there're operations like deletion of old files and creation of new files occurs every day. (this is one of the reason make above file listing so slow, because lots of fragments exist).
3
  • 2
    Do you have possibility to create list with backuped files? If yes, you can save it into file and exclude from rsync: link – dave Aug 28 '12 at 6:22
  • That list can be obtained from the log file, as long as appropriate logging is enabled. A little scripting can extract the entries and build an exclude list. – John Gardeniers Aug 28 '12 at 8:22
  • @japao, I can get file list, however the file size of list file will be large (see question update), 1+ million files per month, and the destination disk will contain about 8-10 months files, can rsync handle such large exclude list? – LiuYan 刘研 Aug 28 '12 at 13:38
1

Maybe git-annex can help you. Take a look at it.

1
  • Looks interesting, especially the location tracking function. But using command line to get file is not friendly to non technician staff. The hard drive disk(s) are kept by workmate whom is not a technician, and when search archived files, she/he'd rather use built in search function of windows explorer. – LiuYan 刘研 Aug 28 '12 at 9:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.