8

I use tar to create a snapshot of different parts of the filesystem on my servers and then ftp that snapshot to an offsite location for archiving.

I would like to only start that operation when something has changed. Some of the backups run on all the system's folders that change very infrequently (i. e. when new software is installed or the configurations are modified).

Whenever a change does happen, I want a complete snapshot. I could produce a list of modified files with find, but I really only need to know if that list's length is 0 or greater. Using find is too slow for that.

I am aware that there is incremental backups and I'm already using rsync in conjunction with ZFS for that in other situations. However, here the backup host is an FTP server (so no rsync), I need complete backups (because the backup archive is used as an image to restore or clone servers) and I want compressed output (so tar is handy).

Edit: Note that I'm not looking for incremental backup (I have that), but rather for a fast (that kinda rules out find and the like) and easy way to decide if a full snapshot would be identical to the last one. Maybe my phrasing wasn't so good. I edited the title now.

Improve this question

12 Answers 12

Reset to default
2

Recent versions of GNU find have the action "-quit", which causes find to immediately stop searching:

— Action: -quit

Exit immediately (with return value zero if no errors have occurred). This is different to ‘-prune’ because ‘-prune’ only applies to the contents of pruned directories, whilt ‘-quit’ simply makes find stop immediately. No child processes will be left running, but no more files specified on the command line will be processed. For example, find /tmp/foo /tmp/bar -print -quit will print only ‘/tmp/foo’. Any command lines which have been built by ‘-exec ... +’ or ‘-execdir ... +’ are invoked before the program is exited.

You could use a find-expression to find files that have changed, and use -quit to stop as soon as you find one. That should be faster than find continuing its scan.

-quit was added in fileutils V4.2.3

Improve this answer
4

GNU tar has a --newer-mtime option, which requires a date argument, which would presumably be the last time you did a backup. Depending on how much work you wanted to restore the filesystem, this could either be the last full backup, in which case you'd need to restore the full dump and the last daily, or you could do it since the last incremental, in which case, you'd need to restore the full dump and every dump after that.

This option does rely on the modification timestamp on the file, so if that has been explicitly changed, then there's a chance your backup will miss it.

Improve this answer
1
  • +1 for usefulness, only I don't really want incremental backup.
    – Hanno Fietz
    Jul 2, 2009 at 16:02
3

The incron utility uses inotify to run commands when filesystem events occur. The configuration file is like a crontab, but instead of times you specify paths and events.

The command could either be your backup script (in which case backup will start almost immediately after the files were modified), or you could have it create some file, and have the backup script check for the existence of that file and then delete it. If the file exists, one of the events occurred since the last run.

Improve this answer
3
  • Neat. I'll give that a shot.
    – Hanno Fietz
    Apr 10, 2012 at 12:47
  • 1
    @HannoFietz recent linux versions support fanotify that notifies for whole filesystems, way more efficient than monitoring each folder and each subfolder separately.
    – user2987828
    Jan 20, 2020 at 15:27
  • github.com/gregghz/Watcher can do it recursively.
    – Konrad Gajewski
    Jan 20, 2020 at 16:57
2

You could always pipe find's output to wc and get an integer count of changed files:

find . -ctime 1 | wc -l

Although David's answer requires fewer code changes :)

Improve this answer
2
  • Well, yes, that's what I thought of, too, but is there anything faster? For my purposes, the find could abort as soon as it found any file that was newer.
    – Hanno Fietz
    Jul 2, 2009 at 16:06
  • Nope, you're pretty much stuck with doing a find, I'm afraid. There is inotify, but I believe that it's only really suitable being notified of a couple of directories or files at most, not an entire filesystem.
    – David Pashley
    Jul 2, 2009 at 16:14
2

This is a little bit of a wild idea, but you could play a little with md5sum and ls.

This idea is to only look at a md5sum of one file, and that that file is a file listing of the dir you are watching. And as long as nothing changes, the md5sum is the same. But if a timestamp is updated the md5sum will change, and you know you need to do a new tar and send it to your ftp server.

We could start with something like this

ls -lR /path/to/dir/ | md5sum > file_list.txt.md5

Then you would need to add a comparison between the old md5 and the current... etc etc

/Johan

Improve this answer
2
  • Or just diff -q the current and previous listings.
    – pgs
    Jul 2, 2009 at 13:45
  • That would work as well.
    – Johan
    Jul 2, 2009 at 14:00
1

tar has a --diff option that will "find differences between archive and file system". If you keep a local copy of the file you uploaded then you could compare them with that.

Improve this answer
2
  • How's that different from looking at the file system with find and screen-scrape the ouput?
    – Hanno Fietz
    Jul 2, 2009 at 13:31
  • I believe the difference is that it's an actual compare, so if the ctime was explicitly changed it wont through off the backup schedule
    – Fire Crow
    Sep 30, 2009 at 9:26
1

You also have the lower case -g option -g, --listed-incremental F create/list/extract new GNU-format incremental backup

I've never played with it but you could script something so test this first on non critical. ;) take a full backup

tar -zcvf /home/backup.tar.gz /backup_dir

then

  NOW=$(date +"%d-%m-%Y")
  i=$(date +"%Hh%Mm%Ss")
  FILE="i.$NOW-$i.tar.gz"

tar -g /home/backup.incremental.txt -zcvf /backup/$FILE /backup_dir
Improve this answer
1

I switched my backup completely to rsnapshot (perlscript, it uses rsync and hardlinks, it can backup remote hosts)

Every night rsync copies just newer files and - thanks to hardlinks - every backupfolder represents the complete data.

rsnapshot is super fast and restore is so easy - give it a try!

Improve this answer
2
  • I have used rsnapshot previously, it's great, I know. However, my problem here is the FTP target, really.
    – Hanno Fietz
    Jul 6, 2009 at 11:46
  • oh, sorry. Then I cannot give you usefull help on this problem :-/
    – ThorstenS
    Jul 6, 2009 at 12:10
1

Radical idea: you can have the system audit the files in question for each access.

This is very verbose in logging terms but would provide you with datestamps on each read/write. Yes, it is similar in concept to Windows NT audit logging. It's probably overkill for your setup, but in the interest of completeness, I'm offing this concept...

You can set up auditing using this short tutorial here.

Pros:

  • catches just about everything, including a history of multiple edits and changes.
  • very fine grain control
  • can selectively audit by file

Cons:

  • increases log spew by a factor of 1 bazillion percent if not tamped down properly to the directories you're concerned about. Use it judiciously.
  • Does not audit data changes, only who did what and when

You can use the ausearch tool to locate changes to files on a per-filename basis. A simple script to iterate over the directories (and subdirectories?) on a per-file basis would allow you to emit changes to a simple file, giving you a list of files that were "touched" in the criteria you specify. You can easily extend this with other filtering options in ausearch for per-user (useful if you have a user account for a service), per-command, etc.

Improve this answer
1
  • +1 for interesting, though I can't see me using it in my scenario.
    – Hanno Fietz
    Jul 6, 2009 at 11:45
1

You could install git and parse the output of "git status" (or maybe the exit codes?) for the directories in question. Git is pretty fast at what it does.

Just make sure to commit the changes, so successive calls to "git status" will show changes.

Another idea would be to use tripwire or some similar tool.

A more brute force approach would be to periodically tar the directories anyway and compare an MD5 of the previous tar. If the directories are large, though, this would not scale too well.

Improve this answer
1

It sounds like what you really want is to trigger a backup process whenever something changes in a directory hierarchy. inotify might be able to help here.

Improve this answer
1
  • Yeah, it would, but I'd still have to program my own little software that would listen to those events and then do something. I'm looking for something that I can just use.
    – Hanno Fietz
    Jul 13, 2009 at 7:52
0

Log the output of ls -UlR --full-time /list/of /folders/to/monitor and pipe it to diff -q that stops comparison if a change is detected between its input and the cached version (first argument).

If diff found a difference, recreate the cached version to take account of last files, then snapshot again the paths like you usually do.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .