48

ZFS is an incredible filesystem and solves many of my local and shared data storage needs. While, I do like the idea of clustered ZFS wherever possible, sometimes it's not practical, or I need some geographical separation of storage nodes. One of the use cases I have is for high-performance replicated storage on Linux application servers. For example, I ...


8

You have two different ways you can do that: The traditional, filesystem-agnostic way that is/was used for the last decades, with tools like rsync or Bacula. There you have tested and (hopefully) stable, large software that can be customized for huge deployments and can be used even if you switch away from ZFS One of the tools that leverage ZFS send/recv. ...


6

No. First: there is not always an archive flag, it is a filesystem-specific thing. For example, in the case of bacula, differential backup means backing up the difference since the last full backup, and incremental backup means backup since the last differential or incremental backups. There is also a big spectrum of backup software which can do only full ...


5

Well, the reason for that is not trust in the backup. It's the amount of work for a restore (among many other considerations that go in a backup scheme). If you have only one full and 300 incremental backups, you need to restore first the full and then every single incremental backup to get to the actual data. If you have weekly full + incremental, you ...


5

A newcomer to the scene is Attic https://attic-backup.org/ We used rdiff-backup for a few years as our primary backup method. It was great for what it did, but created tens/hundreds of thousands of small diff files across the course of a year. Most file systems and disks are going to struggle to deal with a million-plus file count. Backing up our 90GB ...


5

Someone has already written a script that does this, and it has a lot of great features, plus it has been tested by many people for years. It is called "rsnapshot". I highly recommend "rsnapshot". It is easy to configure and will do a 3-day retention, or even more complex retention policies. http://www.rsnapshot.org/


4

Yes. On every rotation (hourly, daily, weekly, monthly), rsnapshot first does a cp -L ("copy" with hard links only), and then runs rsync, which deletes the links and replaces them with new/updated files. If no files change during your whole rotation, you store only one copy of all files, and a bunch of links to the same data on disk. If you enable ...


4

You'll have to schedule backups to UNC paths using the command-line wbadmin tool, as you've noted in your comment. Be warned that backing-up to a UNC path results in only a single generation of backup being stored. Each backup erases the previous backup. This is not the behavior that you'd have been seeing with a locally-attached USB disk. An SMB mounted ...


4

Without application-level quiescing, your backups would be crash-consistent. The recovered VM would look like it had suddenly had its power cut off... For your purposes, it's important to understand how your application deals with sudden power loss or an unplanned hard reset. If it can recover gracefully, then the way Veeam backups are running should ...


4

I'm wondering if anyone can come up with a simpler, more efficient algorithm or add any ideas for a better way of accomplishing this. Yep - use rdiff-backup, rsnapshot, or some other option that you don't need to maintain. I'm not being obtuse here. I realize you said you want to write your own code. Sometimes the correct answer here is not what you want ...


4

Robocopy definitely won't do what you're looking for on its own. Initially I thought about telling you to write a script that dynamically generates the exclusion list, but you're likely to run into command-line length limitations. With that in mind, I think you'd be best off writing a script to create a Robocopy "job" file. Here's a script to create a ...


4

The need of full backups is more obvious when more time goes by. When recovering from the incremental backups you go through all the changes after last full backup in order to recover the current state of every file. How often you should take full backups depends on how long you need to keep incremental data i.e. how distant past versions you may need to ...


3

The way rsnapshot works, on the first run, it creates a full copy. On the next ones, it only stores the files that have changed. If a file hasn't changed, it creates a hard link to it. This means now my files are using double the space (original and snapshot) Yes. You have the working files, and snapshots. The issue is that since I am creating a ...


3

You're attempting to solve a well-practiced problem (in real database systems) using an awkward solution ; this is understandable for most people coming from a background in smaller database systems (and I've done a very similar thing myself with MySQL and slogged through the consequences of bandwidth blowout). You should use PostgreSQL's replication ...


3

What you are trying to do with cp and rsync is create sets of backups with identical files hard-linked to each other. This can be very useful, which is why the rsync developers already though of this and included native functionality directly within rsync to accomplish this. This functionality is provided by the --link-dest option. Basically your rsync ...


3

It would be nice if we had a week, a month and a year that all had a common denominator in days. But we don't. Therefore, there's no correct mathematical answer to your question. It's just a scheme. Pick any day for your yearly backup. It could be within your "grandfather" or "father" cycles, last day of the year, last saturday of the year. What difference ...


2

It seems to me that you're happy with rsnapshot but not with tar. If so, the solution is to replace tar with something that preserves the 'only changes' of rsnapshot. Out of my head, I can thing of two easy options: after rsnapshot, find the differences between the last and previous snapshots to store only those to a tar file. It could be simply diff, ...


2

As the various backup dirs only contain the changed files by the last current backup, the restore procedure is somewhat complex: you need to restore the current full backup and re-apply the various file-level backup, up to the required (past) date. Can I suggest you a better backup method? Please use the --link-dest option combined with proper rotation. ...


2

Bacula doesn't support "volume space reclamation" as you describe, but it does support "Virtual Backups" (what other systems call "Synthetic Backup" which seems to do what you're looking for. Your backup media is still subject to whatever retention periods you've configured, but you could set those to an appropriate period to avoid having extra full backups ...


2

In order to avoid such nightmare, you should use pre-freeze/post-thaw scripts to setup for job which backs this VM up. If needed, check related topics in Veeam forum to ask others about their experience and scripts' examples.


2

Rsync / Rsnapshot are way better tools for this kind of work, especially considering that they give you a "live" snapshot directory where inconsistencies will be limited to some file at most, but they can't bring down the entire backup. Moreover, using hard links, you can have an incremental backup without the inconveniences associated to it. I used this ...


2

Assuming you already increased timeouts, etc, and with no other prior information on your setup, probably there are some very large files on the remote end (each larger than 2Gb) which breaks the flow. Try excluding that/those file(s) through the web interface: Edit Config -> Xfer -> BackupFilesExclude I have been loking for alternatives to backuppc for a ...


2

An alternative to LVM snapshot is to use the datto block driver (aka. dattobd). From the dattobd GitHub page: The Datto Block Driver (dattobd) solves the above problems and brings functionality similar to VSS on Windows to a broad range of Linux kernels. Dattobd is an open source Linux kernel module for point-in-time live snapshotting. Dattobd can be ...


2

No options. You'll need to copy the data over again.


2

You can backup to network shares and the backups are performed incrementally. What is not supported are multiple versions maintained on the network share. You can only access the latest backup version. Backing up to a local disk is dangerous, because you may loose the original data and the backup in case of hardware faults. Its better to choose an iSCSI ...


1

Nicolas I found the same issue that you faces. In my test , I have to upgrade from lxd 2.0.8 to 2.7 up and have to stop lxc (which created or start from the 2.0.x) then start lxc again you will see the backup.yaml and can copy to the destination for use the lxd import (before to import don't forget to zfs set mountpoint and create link container name to ...


1

If the router supports syslog, that can be used to forward logs to a syslog server for long term storage. Even DD-WRT supports syslog.


1

You can only have 9 levels of incrementals, and you will have to start over at level 0. Deleting a level 0 dump will render the dependent incrementals invalid. That said, you can have multiple level 1 dumps. A L1 dump is 'everything since the last L0'. If you have a 1-9 series, a second L1 will contain the cumulative changes in the L2-9 dumps. That second ...


1

I'm sorry, but according to the documentation at https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy: /xf <FileName>[ ...] Excludes files that match the specified names or paths. Note that FileName can include wildcard characters (* and ?). /xd <Directory>[ ...] Excludes directories that match the ...


1

Make dump in sql format. Keep one full copy on local vm, lets say refreshed every day. Then dump new copy and make a diff from full copy. Copy full copy once a day and only diff at other times. To restore you will have to patch full copy with a diff and execute sql file.


Only top voted, non community-wiki answers of a minimum length are eligible