Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 14 down vote favorite
10

It takes forever to back up. Before we can trust btrfs or ZFS to backup incremental snapshots, wouldn't it be nice if there was a daemon that used inotify to keep track of which files had actually changed so backups would run more quickly? Where is this program?

How do I backup my Linux box without having to crawl the whole filesystem every time? I would like a program that would detect the new or changed photos, source code, etc. and queue them to be copied over to my NAS.

share|improve this question
You haven't really given much information here, especially regarding what you are currently using to back up, what type of data. There are several pieces of backup software which use journals to track changes to the filesystem, and then refer to that journal during a backup. What exactly are you trying to back up, to what sort of device/application, what is the nature of the data, and what is your current method for backing it up? – WerkkreW May 13 '09 at 21:18
@WerkkreW, I don't think what he is trying to backup is all that important. I suspect anything that allows for event-based backup would be interesting to learn about. The request isn't unusual, OSX has time machine which is event based. – Zoredache May 13 '09 at 21:52
I still feel the question needs a bit more clarification before it can be answered. – WerkkreW May 13 '09 at 23:09
1  
@Zoredache, of course what I'm backing up is important! If it wasn't important... oh wait ;-) I want to back up my home directory without having an unusable machine for the n hours it would take for rsync to crawl the whole thing, when the only new stuff is a set of photos of my newborn baby. – joeforker May 14 '09 at 1:12
1  
The question seems perfectly clear to me: Mac OS X's Time Machine backup is fast because it monitors fsevents to so it knows where to look when it runs a backup. Linux has inotify, a similar facility to fsevents, and yet Linux backup solutions that try to approximate Time Machine (e.g. dirvish) are miserably slow because they don't take advantage of inotify. Are there any that do? – bendin Aug 20 '09 at 16:31

8 Answers

up vote 9 down vote accepted

I answered my own question with "yum search inotify". It's called lsyncd and it's hosted on google code.

Unfortunately it looks like it always runs a full rsync first, so it still wouldn't help me if my computer was not turned on for more than 14 hours at a time.

Lsyncd uses rsync to synchronize local directories with a remote machine running rsyncd. Lsyncd watches multiple directories trees through inotify. The first step after adding the watches is to rsync all directories with the remote host, and then sync single file by collecting the inotify events. So lsyncd is a light-weight live mirror solution that should be easy to install and use while blending well with your system. See lsyncd --help for detailed command line options.

share|improve this answer
That's an interesting link, I had even thought of implementing that myself. But why does it say on the page that it will retransfer large files for each change? I thought rsync itself would already avoid that? – Hanno Fietz Jul 2 '09 at 13:06
rsync still has to re-read the entire file on both ends to transfer it as efficiently as it can. The lsyncd documentation intends to say that this may not be efficient for large files. For large files a block-level replication scheme would be more appropriate. – joeforker Jul 6 '09 at 13:15
up vote 3 down vote

There's a new system called fsnotify that's designed to solve the deficiencies of inotify which was introduced to solve the problems of dnotify. fsnotify lets you watch an entire filesystem without much fuss. Hopefully fsnotify will help solve all our future Linux backup problems.

share|improve this answer
up vote 2 down vote

You can hack something with incron. 

 /path1    IN_CLOSE_WRITE     rsync  -au $@/$#  backuphhost:/path

share|improve this answer
incron can't watch directories recursively – joeforker May 18 '09 at 13:12
well, duh! it seemed very useful :-S – hayalci May 18 '09 at 18:13
up vote 2 down vote

Based on research (not testing) it seems like inotify can't handle the very large number of files on most systems and/or is very slow doing so. The thread at http://www.pubbs.net/kernel/200905/109416/ was the most useful, It pointed to a new linux feature that seems to be in or headed towards mainline, fsnotify, that is in linux 2.6.31 and later.

share|improve this answer
up vote 2 down vote

Lsyncd syncs the whole watched tree on startup, because for 99% of cases this is the sensible thing to do. You do want the directory on the target host you have on the local host, otherwise syncs might fail and you want to sync things you missed while turned off. However, if you know what you are doing, you can turn off startup syncing, just set sync{..., startup=false} in the Lsyncd config file.

regarding inotify, its not the number of files but the number of directories that eat up ressources. One directory is one watch, regardless how many files it contains.

fanotify building on fsnotify like inotify looked very promising for people watching tons of directories, but currently as of Linux 2.6.37 fanotify does not report rename (move) events at all, making it unusable for a job like this :-(

share|improve this answer
Not even as a delete followed by a create? – joeforker Dec 13 '10 at 19:42
1  
not in my tests, nope :-( fanotify was designed with the virus shield in mind, and IMHO is tailored right now exactly to the minimum it requires. – axkibe Dec 13 '10 at 21:19
up vote 1 down vote

You're kind of missing a better way, even though you keep hinting at it.

NTFS has a change journal specifically for use by backup software. It tracks all the files that have changed sequentially, in a circular log. Thus if you are backup software and you know that you backed up all the changes on this filesystem last time up to change number 1234, then all you need to pick up this time (presuming an incremental backup) is the changes marked higher than number 1234. You don't need to scan the whole tree, even if you haven't been running for weeks, and you don't have to take shortcuts (the way we commonly do with rsync) like looking only at file mod time instead of file contents.

The limitation of a change journal approach is that your change journal has to be large enough to track all the changes between your two backups, or the backup software will have to resort to older approaches. In practice with NTFS it's not uncommon (especially for fileservers with large numbers of small files and a high change rate) to have to increase the change journal from its default size -- however, it's not that big to start with, this doesn't take a lot of space.

If you really have a big file tree and you want to do the same approach with inotify+rsync, just re-invent the NTFS change journal by having a small script listen for inotify events and log them to a circular log, and then feed that to rsync instead of having it walk the trees.

share|improve this answer
up vote 1 down vote

I have spent 6 months looking for the best solution to perform what you're trying to do; efficiently backup to a NAS. After the initial sync, all else is smooth as butter. The latest version of Lsyncd works quite well. I've documented what I've done in the link below. Just substitute your folder values. Hope this helps:

https://docs.google.com/document/d/1XpqM5h5YMwuQqzdknyDDnjcQVYGjAsyAxfYprqSnhd0/edit

share|improve this answer
up vote 0 down vote

there 's a new system used in asia,called sersync which will instead of inotify-tool+rsync resolution http://code.google.com/p/sersync/ it is very easy to use.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.