Hot answers tagged rsnapshot
28
No.
"rm -rf" does a recursive depth-first traversal of your filesystem, calling unlink() on every file. The two operations that cause the process to go slowly are opendir()/readdir() and unlink(). opendir() and readdir() are dependent on the number of files in the directory. unlink() is dependent on the size of the file being deleted. The only way to make ...
8
You can almost certainly ignore /proc, /dev, /tmp and /var/tmp.
A good case can be made for ignoring /var/log (and any other logging directories), /var/cache if you have it, and possibly portions of /var/db (though you have to be careful with /var/db: Sometimes really important stuff gets put there...)
Beyond that you probably want to do a backup, wait a ...
7
Another option is to set the directory to be immutable. To do this you need to run the following command with the mount point unmounted.
chattr +i /backups
I do this on any directory that is intended to only be a mount point just to prevent this sort of thing. Because there are situations where you're not in a position to add a check to see if something ...
6
No, this is not correct. If you have multiple hardlinks to a file, it doesn't matter which one originally created the file, the file will only be deleted if the last link to the file is deleted (see the difference between a hard link as used by rsnapshot and a symbolic link) In the case of rsnapshot this means that every backup directory is self-contained ...
4
TL;DR: no.
It depends on what you define an "initial backup".
You first create a backup (hourly.0), which has all the files from today.
On the next iteration, it "copies" the files (cp -L, just copies the links to the data), to hourly.1 folder.
If all the files are the same as before, rsync won't write anything, so you have one block of data for a file ...
4
You are reinventing the wheel. You should try using rsync. rsync will build the file list for you, and uses an amazing algorithm that is very fast, even over slow links, or encrypted connections that are slower from the overhead.
Very easy to run as well rsync -vvarP root@server1:/var/www/ root@server2:/var/backup/
4
On Linux, btrfs is your main option for snapshots within a filesystem, but is not yet stable and doesn't have an fsck yet.
ZFS is another option, from Solaris. While you could in theory use this under Linux, it's mostly implemented in userland only (though there is some work on a kernel-based implementation) and is not in standard Linux distributions.
...
4
If you are doing the rm at effectively at the file level then it will take a long time. This is why block based snapshots are so good:).
You could try splitting the rm into separate areas and trying to do it in parallel however I might not expect it to make any improvement. XFS is known to have issues deleting files and if that is a large part of what you ...
4
At the point rsync seems to hang, is it hung or just busy? Check for cpu activity with top and disk activity with iotop -o.
It could be busy copying over a large file. You would see this in iotop or similar, or in rsync's display if you ran it with the --progress option.
It could also be busy scanning through lists of inodes to check for linked files. If ...
4
ext4 is fine.
The "sync,dirsync" options make data and metadata updates synchronous, which is a major negative effect on performance on most workloads. Removing these options will very likely increase performance, but you have to remember to unmount the device before yanking the cable, otherwise you might lose data (presumably this is why the options were ...
3
My answer, which I give from hard-earned experience, is: Don't do this. Don't try to copy a directory hierarchy that makes heavy use of hard links, such as one created using rsnapshot or rsync --link-dest or similar. It won't work on anything but small datasets. At least, not reliably. (Your mileage may vary, of course; perhaps your backup datasets are much ...
3
You can probably skip /var/run, but that directory holds so little data, it's probably not worth skipping.
Sure, you can back up to /var/backup (or whatever location you wish, really). Just be careful exclude your /var/backup from your backup sets otherwise you'll get into a recursive backup situation, which isn't good.
Additionally, I assume that ...
3
I'm a big fan of snapshots for backups, but I typically use them in the manner you'll find described for LVM snapshots -- Make a snapshot, mount it somewhere, back up the snapshot with your favorite backup tool (I like Bacula), then get rid of the LVM snapshot. Ideally this backup goes somewhere off-site
This dovetails very well with rsnapshot: Your ...
3
I don't think this is the most likely explanation, but having read the trouble you're having with rsync, it's just possible that you're suffering from a duplex mismatch on one or both of the NIC-switch connections.
Try doing a netstat -in on both servers, and check the error counts on transmission. Non-zero TX-errors often signal a duplex mismatch, and one ...
3
The rsync command's -H (or --hard-link) option will, in theory, do what you are trying to accomplish, which is, in brief: to create a copy of your filesystem that preserves the hard linked structure of the original. As I mentioned in my answer to another similar question, this option is doomed to fail once your source filesystem grows beyond a certain ...
3
Put the following lines into filter.txt:
+ */
+ *.php
+ *.txt
+ *.html
+ **/_dir*.jpg
- **/_*.jpg
+ *.jpg
- *
Then run rsync like this:
rsync -a --include-from=filter.txt /path/to/source/ /path/to/dest
Explanation of filter.txt:
First of all, note that the ordering of the filter rules is very important, since rsync evaluates them in order and applies ...
3
With a default configuration (save for changing the backup location to /smallbup) and rsnapshot 1.3.1
svr-backup-01 ~ # lvcreate --name smallbup --size 32M RAID
Logical volume "smallbup" created
svr-backup-01 ~ # mkfs.ext4 -q -m 0 /dev/RAID/smallbup
svr-backup-01 ~ # mkdir /smallbup
svr-backup-01 ~ # mount /dev/RAID/smallbup /smallbup/
svr-backup-01 ~ # ...
3
Ensure you are also monitoring your filesystem free space, monitor system logs for critical / severe messages, SMART output for your disks, network and backup services (ssh / rsync).
Regarding verifying your backups, you may want to setup your webapps environment in parallel and recover your backup periodically. Your backups are as good as your recovery.
3
The rsnapshot man page seems to imply that you add the per-backup-point arguments as a comma separated list after the destination, the example they give is
backup /var/ localhost/ one_fs=1
This is the same as the other examples, but notice the fourth column. This is how you specify per-backup-point options to over-ride global settings. This extra ...
3
This behaviour is actually controlled by rsync's --relative flag. Quoting the rsync manual:
-R, --relative
Use relative paths. This means that the full path names specified on
the command line are sent to the server rather than just the last
parts of the filenames. This is particularly useful when you want to
send several different directories ...
2
Short answer: replace this with
retain manual 3
and run rsnapshot manual whenever you'd like to do a backup.
The default configuration defines three different “backup levels.” The idea is that you run rsnapshot hourly every few hours, rsnapshot weekly every week, and rsnapshot monthly every month (usually from cron jobs). When rsnapshot ...
2
There is a perl script that comes bundled with rsnapshot called rsnapreport.pl, when run it outputs something like this :
SOURCE TOTAL FILES FILES TRANS TOTAL MB MB TRANS LIST GEN TIME FILE XFER TIME
----------------------------------------------------------------------------------------------------
myhost:/ 5176 ...
2
First of all you should move the backup script to the remote system. Bear in mind that the file is being executed on the remote system here, not on the local one (the one that runs rsnapshot). Then once the program is executed on the remote system you should move the files back to the local system.
Something like this should work :
backup_script ...
2
rsnapshot configuration files are tab delimited. Make sure you include tabs between the backup_scrip, the ssh command, and the backup storage. If that does not work try running rsnapshot configtest to identify the problem.
Answer updated
It is more convenient to transform your backup line to the following format:
backup_script /dir/yourscript.sh ...
2
So yes, you must provide the argument in the rsnapshot.conf file :
rsync_long_args --ignore-existing
If you want to keep the default rsync_long_args for rsnapshot (you probably want). You must put them on the same line like this :
rsync_long_args --delete --numeric-ids --relative --delete-excluded --ignore-existing
The example above is ...
2
you have to understand how rsnapshot generates the backups.
First of all: Only the first interval you are defining takes the snapshot all other intervals just move old snapshots.
In your example:
interval halfhourly 4
first moves the old snapshots from halfhourly.n to halfhourly.n+1. Then it takes a new snapshot in halfhourly.0.
When you run the other ...
2
Have you tried, http://duplicity.nongnu.org/ ? It supports a lot of things to backup on including Amazon S3. From the features page,
Currently local file storage, scp/ssh, ftp, rsync, HSI, WebDAV, Tahoe-LAFS,
and Amazon S3 are supported, and others shouldn't be difficult to add.
rsnapshot isn't an actively developed project any more (last release 1.2.0 ...
2
To address your points:
This just means that all exclusion rules you specify must be files, and not directories. Since rsync doesn't distinguish on type, you've got to be careful with your rules (and file naming conventions).
Use:
+ **/*.php
+ **/*.txt
+ **/*.html
Use:
+ **/*.jpg
- **/_*.jpg
I have no idea what you mean.
2
The only folders you should need are /var/www and /var/lib/mysql to get your website and data. And backup /etc/httpd to get your apache configuration, if necessary. See here for a discussion on backing up /var/lib/mysql versus using mysqldump.
If you can use an lvm snapshot to backup from, that would be even better, but be sure to destroy the snapshot as ...
2
If you use rsync with -H to copy the entire directory (i.e. the parent of your daily.n directories) it will preserve the hard links when copying to a new location.
I don't know of a nice way to only copy individual snapshots to an external disk and preserve the hard links - hard links only work on the same filesystem.
Only top voted, non community-wiki answers of a minimum length are eligible
