Hot answers tagged

65

I have had very good results using tar, pigz (parallel gzip) and nc. Source machine: tar -cf - -C /path/of/small/files . | pigz | nc -l 9876 Destination machine: To extract: nc source_machine_ip 9876 | pigz -d | tar -xf - -C /put/stuff/here To keep archive: nc source_machine_ip 9876 > smallstuff.tar.gz If you want to see the transfer rate just ...


21

I'd stick to the rsync solution. Modern (3.0.0+) rsync uses incremental file list, so it does not have to build full list before transfer. So restarting it won't require you to do whole transfer again in case of trouble. Splitting the transfer per top or second level directory will optimize this even further. (I'd use rsync -a -P and add --compress if your ...


15

Set up a VPN (if its internet), create a virtual drive of some format on the remote server (make it ext4), mount it on the remote server, then mount that on the local server (using a block-level protocol like iSCSI), and use dd or another block-level tool to do the transfer. You can then copy the files off the virtual drive to the real (XFS) drive at your ...


15

Using the Debian archive you shouldn't have an apt source line referring to wheezy-updates. This particular (sub-)distribution is not existing in the archives. If you just remove the lines: deb http://security.debian.org/ wheezy/updates main deb http://ftp.fr.debian.org/debian wheezy-updates main you should be able to run your apt-get commands again. ...


12

Tape. Simple like that. Quantum has a SuperSTore system that can handle way more than that and I have seen them for less than your 5000 price point - new. The good thing is that you can pull tapes out for storage so scaling this is going to be quite cost efficient, and tapes last.


11

You could use SquashFS for such archives. It is designed to be accessed using a fuse driver (although a traditional interface exists) compressed (the larger the block size, the more efficient) included in the Linux kernel stores UIDs/GIDs and creation time endianess-aware, therefore quite portable The only drawback I know of is that it is read-only. http:/...


9

If the old server is being decommissioned and the files can be offline for a few minutes then it is often fastest to just pull the drives out the old box and cable them into the new server, mount them (back online now) and copy the files to the new servers native disks.


9

Add the security updates: deb http://archive.debian.org/debian wheezy main deb http://archive.debian.org/debian-security wheezy/updates main


5

Thorbjørn Ravn Anderser is right. GNU tar creates "seekable" archives by default. But it does not use that information when it reads these archives if -n option is not given. With -n option I just extracted 7GB file from 300GB archive in time required to read/write 7GB. Without -n it took more than hour and produced no result. I'm not sure how compression ...


4

First, I would advise avoiding Glacier. It sounds good, until you crunch the costs on actually restoring a large amount of data. This is an unofficial calculator you can use to calculate Glacier storage and retrieval costs, and judge for yourself. Restoring terabytes of data from Glacier is a pretty unattractive prospect. Second, I would advise that for ...


4

Some technical comparisons among v7, ustar and pax formats: v7 The format before POSIX.1-1988. Maximum length of a file name is 99 characters. (100 bytes minus a terminating null byte.) Maximum length of a link target is 99 characters. File types allowed: regular file (typeflag '\0'), directory, hard link (typeflag 1), symbolic link (typeflag 2). ...


4

Expanding and building on top of @dvb's answer, you can set the deepness level with extended regular expressions: zipinfo file.zip | egrep "^([^/]*/?){<deepness-level>}$"


4

If like me you are dealing with an embedded wheezy system and just need to make something work, here are the complete steps that worked for me. The original sources no longer exist which causes the 404 error (not found), so we need to modify the source list. Remove the original line add the two archive lines. vi /etc/apt/sources.list deb http://archive....


4

We have implemented the solution. The QLC drives appear to be fine for the use that we make. However RAIDZ2 showed to be non-practical: The combination ashift=12 with 16K recordsize (appropriate recordsize for our DB) leads to a high price paid on parity. Using RAIDZ2 we had two 4K parity blocks written for 16K actual data. One third of the storage was used ...


3

Use mbuffer and if it is on a secure network you can avoid the encryption step.


3

(Many different answers can work. Here is another one.) Generate the file list with find -type f (this should finish in a couple of hours), split it to small chunks, and transfer each chunk using rsync --files-from=....


3

Unfortunately, it's just one of the facts of user profiles/network shares that users are wasteful/disorganized/etc. If you're determined to use some kind of tool or script to tidy up, or to delete things in "unofficial" locations, make certain that management is on your side and will back you up. Ditto if you decide to implement some kind of disk quota. ...


3

Disclaimer: this answer doesn't contain the solution about how to make postfix bcc email twice. I'm just decipher the content of your maillog line. Postfix part pre-content_filter Feb 3 13:30:25 email1mail postfix/submission/smtpd[3319]: connect from somewhere[xxx.xxx.xxx.xxx] Feb 3 13:30:25 email1mail postfix/submission/smtpd[3319]: 47F943003E: client=...


3

You are actually using the right option, but duplicity will ignore it due to the equals sign =. So rather than using --archive-dir=/volume4/duplicity/cache/ --name backup you must use: --archive-dir /volume4/duplicity/cache/ --name backup


3

I meant to mention that DF32 has an option to save a "raw" image, that does not have the DFI file header, it's just the sector data, and is mountable in a VM. And I found the XP version does work in Win7, I successfully read a set of install diskettes from 1992, using a USB-connected diskette drives. (Yes I was amazed!) :-) http://mcgintys.net/diskfactory/...


3

Have you considered sneakernet? With that, I mean transfering everything onto the same drive, then physically moving that drive over. about a month ago, Samsung unveiled a 16 TB drive (technically, it's 15.36 TB), which is also an SSD: http://www.theverge.com/2015/8/14/9153083/samsung-worlds-largest-hard-drive-16tb I think this drive would just about do ...


3

While typing my question and reading the man page more thoroughly, it came to me that the answering flag is -g, so to extract from an archive a single directory into the current one, use: dar -x archive-name -g dir/to/extract According to the man page, dar features six different parameters for file selection, namely -I vs -X, -P vs -g and -[ vs -]. The ...


3

Due to how HP drives report their SMART info, the provided data are not tremendously useful. That said, attribute 173 should be the worst-case erase count (ie: wear) of NAND blocks. With only 26 max erase cycles after 6 months, your SSD should be good for 3000 / 26 / 2 = ~57.7 years. This is clearly an exaggeration, as much before that you will need to ...


3

journald rotates its own files, so it's safe to copy the old files (which always have an @ in the filename) elsewhere, compress them, or do whatever you want with them. If it's necessary to look into them, journalctl can be pointed at specific journal files with the --file command line option. You should also see journald.conf for options to limit journald's ...


2

FIrst, I would use one of the tools Braiam described to put the the packages from the first server in a local repository on the second server. Next, I would run dpkg --get-selections on the first server, and feed it's output to dpkg --set-selections on the second server; that will mark software to be installed. FInally, I would run apt-get dselect-upgrade on ...


2

You can simply copy all the .deb packages and install them with dpkg: sudo dpkg -i *.deb There isn't any danger unless you forget some package or have the same package but with different versions. I would use APTonCD, or Apt-mirror, or AptMedium as the safest methods. There are a bunch of them in either: https://help.ubuntu.com/community/AptGet/Offline/ ...


2

I have the same problem, using systemrescuecd, which is a livecd. I had to symlink /root/.cache/duplicity to a tempdir on a mounted harddrive as a solution.


2

You are using RedHat Linux, so this wouldn't apply, but as another option: I've had great success using ZFS to hold millions of files as inodes aren't an issue. If that was an option for you, you could then take snapshots and use zfs to send incremental updates. I've had a lot of success using this method to transfer as well as archive data. ZFS is ...


2

If there is any chance to get high success ratio when deduplication, I would use something like borgbackup or Attic. If not, check the netcat+tar+pbzip2 solution, adapt the compression options according to your hardware - check what is the bottleneck (CPU? network? IO?). The pbzip2 would nicely span across all CPUs, giving better performance.


2

You can use the 7z (7zip) archive/compression format if you have access to the p7zip-full package. On Ubuntu you can use this command to install it: $ sudo apt-get install p7zip-full To create an archive you can use 7z a <archive_name> <file_or_directory> and if you do not want to compress the files and want to just "store" them as-is, you can ...


Only top voted, non community-wiki answers of a minimum length are eligible