Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

It takes forever to back up. Before we can trust btrfs or ZFS to backup incremental snapshots, wouldn't it be nice if there was a daemon that used inotify to keep track of which files had actually changed so backups would run more quickly? Where is this program?

How do I backup my Linux box without having to crawl the whole filesystem every time? I would like a program that would detect the new or changed photos, source code, etc. and queue them to be copied over to my NAS.

share|improve this question
    
You haven't really given much information here, especially regarding what you are currently using to back up, what type of data. There are several pieces of backup software which use journals to track changes to the filesystem, and then refer to that journal during a backup. What exactly are you trying to back up, to what sort of device/application, what is the nature of the data, and what is your current method for backing it up? –  WerkkreW May 13 '09 at 21:18
    
@WerkkreW, I don't think what he is trying to backup is all that important. I suspect anything that allows for event-based backup would be interesting to learn about. The request isn't unusual, OSX has time machine which is event based. –  Zoredache May 13 '09 at 21:52
    
I still feel the question needs a bit more clarification before it can be answered. –  WerkkreW May 13 '09 at 23:09
1  
@Zoredache, of course what I'm backing up is important! If it wasn't important... oh wait ;-) I want to back up my home directory without having an unusable machine for the n hours it would take for rsync to crawl the whole thing, when the only new stuff is a set of photos of my newborn baby. –  joeforker May 14 '09 at 1:12
3  
The question seems perfectly clear to me: Mac OS X's Time Machine backup is fast because it monitors fsevents to so it knows where to look when it runs a backup. Linux has inotify, a similar facility to fsevents, and yet Linux backup solutions that try to approximate Time Machine (e.g. dirvish) are miserably slow because they don't take advantage of inotify. Are there any that do? –  bendin Aug 20 '09 at 16:31
add comment

7 Answers 7

up vote 11 down vote accepted

I answered my own question with "yum search inotify". It's called lsyncd and it's hosted on google code.

Unfortunately it looks like it always runs a full rsync first, so it still wouldn't help me if my computer was not turned on for more than 14 hours at a time.

Lsyncd uses rsync to synchronize local directories with a remote machine running rsyncd. Lsyncd watches multiple directories trees through inotify. The first step after adding the watches is to rsync all directories with the remote host, and then sync single file by collecting the inotify events. So lsyncd is a light-weight live mirror solution that should be easy to install and use while blending well with your system. See lsyncd --help for detailed command line options.

share|improve this answer
    
That's an interesting link, I had even thought of implementing that myself. But why does it say on the page that it will retransfer large files for each change? I thought rsync itself would already avoid that? –  Hanno Fietz Jul 2 '09 at 13:06
    
rsync still has to re-read the entire file on both ends to transfer it as efficiently as it can. The lsyncd documentation intends to say that this may not be efficient for large files. For large files a block-level replication scheme would be more appropriate. –  joeforker Jul 6 '09 at 13:15
add comment

There's a new system called fsnotify that's designed to solve the deficiencies of inotify which was introduced to solve the problems of dnotify. fsnotify lets you watch an entire filesystem without much fuss. Hopefully fsnotify will help solve all our future Linux backup problems.

share|improve this answer
add comment

You can hack something with incron.

 /path1    IN_CLOSE_WRITE     rsync  -au $@/$#  backuphhost:/path

share|improve this answer
    
incron can't watch directories recursively –  joeforker May 18 '09 at 13:12
    
well, duh! it seemed very useful :-S –  hayalci May 18 '09 at 18:13
add comment

Based on research (not testing) it seems like inotify can't handle the very large number of files on most systems and/or is very slow doing so. The thread at http://www.pubbs.net/kernel/200905/109416/ was the most useful, It pointed to a new linux feature that seems to be in or headed towards mainline, fsnotify, that is in linux 2.6.31 and later.

share|improve this answer
add comment

Lsyncd syncs the whole watched tree on startup, because for 99% of cases this is the sensible thing to do. You do want the directory on the target host you have on the local host, otherwise syncs might fail and you want to sync things you missed while turned off. However, if you know what you are doing, you can turn off startup syncing, just set sync{..., startup=false} in the Lsyncd config file.

regarding inotify, its not the number of files but the number of directories that eat up ressources. One directory is one watch, regardless how many files it contains.

fanotify building on fsnotify like inotify looked very promising for people watching tons of directories, but currently as of Linux 2.6.37 fanotify does not report rename (move) events at all, making it unusable for a job like this :-(

share|improve this answer
    
Not even as a delete followed by a create? –  joeforker Dec 13 '10 at 19:42
1  
not in my tests, nope :-( fanotify was designed with the virus shield in mind, and IMHO is tailored right now exactly to the minimum it requires. –  axkibe Dec 13 '10 at 21:19
add comment

I have spent 6 months looking for the best solution to perform what you're trying to do; efficiently backup to a NAS. After the initial sync, all else is smooth as butter. The latest version of Lsyncd works quite well. I've documented what I've done in the link below. Just substitute your folder values. Hope this helps:

https://docs.google.com/document/d/1XpqM5h5YMwuQqzdknyDDnjcQVYGjAsyAxfYprqSnhd0/edit

share|improve this answer
add comment

there 's a new system used in asia,called sersync which will instead of inotify-tool+rsync resolution http://code.google.com/p/sersync/ it is very easy to use.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.