Hot answers tagged

114

You need to take a step back and stop thinking "I've got 20TB on my NAS I need to back up!" and develop a storage strategy that takes into account the nature of your data: Where is it coming from and how much new data are you getting? (you've got this in your question) How is the data used once you have it? Are people editing the pictures? Do you keep the ...


65

Thanks to a tip by Wayne Davison, I use the --stats option for backup: rsync -am --stats src/ dest/ Nice little summary at the end, e.g. Number of files: 6765 Number of files transferred: 0 Total file size: 709674 bytes Total transferred file size: 0 bytes (10 more lines)


56

I go a step further and always set the attributes of my mountpoint directories to immutable using chattr. This is accomplished with chattr +i /mountpoint (with the mount unmounted). This would error-out on new write activity and also protects the mount point in other situations. But I suppose you could use the mountpoint command, too ;)


54

All your suggestions currently have one thing in common: the backup source does the backup and has access to the backup destination. Whether you mount the location or use tools like SSH or rsync, the source system somehow has access to the backup. Therefore, a compromise on the server might compromise your backups, too. What if the backup solution has ...


42

No, it's not. What happens when your filesystem or RAID volume gets corrupted? Or your server gets set on fire? Or someone accidentally formats the wrong array? You lose all your data and the not-real-backups you thought you had. That's why real backups are on a completely different system than the data you're backing up - because backups protect ...


40

What you describe is essential a geographically distributed RAID and a RAID was never a backup. Online sync usually means everything you do on the primary storage gets immediately replicated to the backup system, including operations like the deletion of (all) snapshots and/or volumes by an attacker or simply an admin error.


39

ZFS is an incredible filesystem and solves many of my local and shared data storage needs. While, I do like the idea of clustered ZFS wherever possible, sometimes it's not practical, or I need some geographical separation of storage nodes. One of the use cases I have is for high-performance replicated storage on Linux application servers. For example, I ...


35

To communicate to s3 you need to have 2 things IAM user credentials who has read-write access to s3 bucket. A client like aws-cli for bash, boto library for python etc. once you have both, you can transfer any file from your machine to s3 and from s3 to your machine. Below is the example for aws-cli. to sync all files in a folder aws s3 sync ...


33

Because the transfer speeds out of the ESXi console are purposefully limited. Because this isn't scalable in any way. Because you'd have to drop a statically-compiled rsync binary onto the ESXi host. Because the VMs, the VMDKs, their ramdisk files and other components can change enough to make rsync a losing proposition... do you really want to re-sync a ...


30

You are seriously over-engineering this. Badly. Here's some pseudocode: Every day: make a backup, put into daily directory remove everything but the last 7 daily backups Every week: make a backup, put into weekly directory remove everything but the last 5 weekly backups Every month: make a backup, put into monthly directory remove everything but the ...


29

Well, first of all, you should always have an off-site copy of your data. Not just because there might be a fire, but what would you do in the event of a natural disaster, or similar event making your building inaccessible? Without a copy of your data off site, you're just hosed. So, that said, there are two ways to have an off site copy of your data. ...


29

Rsync does a one way sync, however it's up to you to decide which way the sync goes. Rsync command syntax is the following: rsync [OPTION...] SRC... [DEST] Note that you specify sync from source to destination. Source and destination can be any local or remote path. For example if you want to copy files from your server to your laptop you do: rsync [...


26

This link shows what one can do to speed up restoring process. http://dev.mysql.com/doc/refman/5.5/en/optimizing-innodb-bulk-data-loading.html One can put put the commands at the top of the dump file SET @OLD_AUTOCOMMIT=@@AUTOCOMMIT, AUTOCOMMIT = 0; SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS = 0; SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS,...


25

You're right. Since you can have multiple containers with volumes on their own, you need to keep track which volume corresponds to which container. How to do that depends on your setup: I use the name -data for the data container, so it's obvious to which container a image belongs. That way it can be backed up like this: VOLUME=`docker inspect $NAME-data | ...


23

/var/opt/gitlab/backups by default. https://gitlab.com/gitlab-org/gitlab-ce/blob/a64d0f79173301dd86db073e9b6a329b56554dbd/doc/raketasks/backup_restore.md says that: This archive will be saved in backup_path (see config/gitlab.yml) Then https://gitlab.com/gitlab-org/omnibus-gitlab/blob/9e97671d8dee3ea6a7bbb99347ad279b65689f28/files/gitlab-cookbooks/...


22

Design the right way and you'll minimize the chances of data loss of ZFS. You haven't explained what you're storing on the pool, though. In my applications, it's mostly serving VMWare VMDK's and exporting zvols over iSCSI. 150TB isn't a trivial amount, so I would lean on a professional for scaling advice. I've never lost data with ZFS. I have experienced ...


21

There is a simple way to test if your shell is clean, for an ssh connection: run a command from the ssh connection, rather than starting an interactive shell. The false command will immediately terminate without producing any output, so it is a good test: bash$ ssh remotehost false Enter passphrase for key '/home/user/.ssh/my_private_key': bash$ If that ...


21

We build backup systems for one purpose: To enable restores. Nobody cares about backups; they care about restores. There are three reasons one might need to restore file(s): Accidental file deletion, hardware failure, or archival/legal reasons. A "complete" backup system would enable you to restore files in all of these scenarios. For accidental file ...


21

You can try rclone. Setup (docs): rclone config Sync (docs): rclone sync /my/local/folder s3service:bucket-name


20

You can periodically run crontab -l > my_crontab.backup to backup the crontab into file.


20

Back them up on Amazon glacier. Cheap storage for things that you don't need to access often. It will cost you 1cent per GB per month. As a downside, it might be pricey to fetch your data back in case of failure.


20

This is possible with s3cmd 1.5+ (link): $ mysqldump ... | s3cmd put - s3://bucket/file-name.sql


19

I had the same problem after updating my Synology NAS to DSM 4.1. I also do rsync over SSH. In my situation using rsync user root@ip also worked but left the files on the server with owner root which I did not want. I found in the NAS admin userinterface, that for some reason in ControlPanel->Users->MyRSyncUser->Edit->"Priviliges setup" there was no read/...


19

According to the pg_dump documentation the custom format is: the most flexible output format in that it allows manual selection and reordering of archived items during restore. This format is also compressed by default You can also select the compression level with the -Z option. For simplicity I would definitely go for pg_dump -F c over piping to ...


19

I googled a few examples of LTO-6 tape drives, and they all seem to have basically the same LEDs, although their arrangement varies by OEM and form factor: encryption cleaning required tape (error) drive (error) ready There is no power button LED, so my guess is that the power button and its LED are not integral parts of the drive mechanism itself, but ...


18

Consider this situation: You drop a box with 6 months worth of backups down the stairs. If these are tapes in their cases, you pick the box back up and continue on your way. If they are disks, you just lost 6 months worth of backups and you'll probably be job hunting. It's common practice for couriers to take tapes offsite daily and then return them later. ...


18

This is definitely possible with the --show-session-key and --override-session-key options. First you need the beginning of your encrypted file. This is where the encrypted session key is stored. root@qwerty:~/gpg# head -c 1024k bigfile.gpg > head.gpg Then copy it to your workstation and retrieve the session key PS C:\Users\redacted\Downloads> gpg --...


18

In your case it is the file level encryption that is preventing compression. Encryption tries to make the data stream look as much as random "noise" as possible. Compression tries to increase the data "density" which has a similar effect of limiting further compression.


17

there's a --delete option. without this, it won't delete old files.


16

The option you want is gitlab_rails['backup_keep_time']. I had to gitlab-ctl reconfigure after setting it for it to have any effect. Note that it will only affect local backups, not Amazon-AWS S3 if you're using that. There was an issue with them being ignored, but I'm not sure what versions it affects. Version 6 is pretty old. If you're on the omnibus, ...


Only top voted, non community-wiki answers of a minimum length are eligible