Tag Info

Hot answers tagged

64

mysqldump --opt <database> | gzip -c | ssh user@wherever 'cat > /tmp/yourfile.sql.gz' You can't use tar in a pipe like this, and you don't need it anyway, as you're only outputting a single file. tar is only useful if you have multiple files.


54

tar has 3 types of syntax (according to this ): long options (--file) short options (-f) old options (f) For the old option syntax, all the letters must follow "tar" and must all fall into a single clump with no spaces. The order of the letters doesn't really matter, so long as the arguments to those letters follow the same order after the clump of ...


51

I'd be tempted to rsync it over myself - it does compression and handles link loss well.


31

Instead of using tar to write to your local disk, you can write directly to the remote server over the network using ssh. server1$ tar -zc ./path | ssh server2 "cat > ~/file.tar.gz" Any string that follows your "ssh" command will be run on the remote server instead of the interactive logon. You can pipe input/output to and from those remote commands ...


26

Putting every file into a separate tar file doesn't make any sense in this scenario. You can use gzip to compress them directly: gzip * will result in file1.out.gz, file2.out.gz etc. You would use tar only if you would need a compressed archive as a single file. If you ineed need a tar archive for every file, you can create it like so: for i in *; ...


21

Don't tar. Use rsync -av to preserve permissions while transfering the files. Though like tar, this does not preserve selinux context. Not that I would consider that important though.


19

pigz is a parallel version of gzip. Although it only uses a single thread for decompression, it starts 3 additional threads for reading, writing, and check calculation. Your results may vary but we have seen significant improvement in decompression of some of our data sets. Once you install pigz the tar file can be extracted with: pigz -dc target.tar.gz ...


16

Summing up previous answers and adding some important information: When creating archives, tar will always preserve files' user and group ID, unless told otherwise with --owner=NAME, --group=NAME. But still there will always be a user and group associated with each file. GNU tar, and perhaps other versions of tar, also store the user and group names, ...


15

If you install version 3+ of rsync it will do a rolling list of files to transfer and won't need to keep the entire file list in memory. In the future you probably want to consider hashing the filenames and creating a directory structure based on parts of those hashes. You can see this answer to get an idea of what I mean with the hashing.


15

What you're looking for is the -C option: $ man tar <snip> -C, --directory DIR change to directory DIR Usage, per your example: $ tar xvzf archivename.tar.gz -C /tmp/testdir/ sampledir/ Also, you may be interested to know that modern versions of tar don't require the z flag to decompress gzipped files; they'll notice it's a .gz and Do The ...


14

I don't know if this one is "the best", but you can try 7-Zip


14

Use the split command. split -b 22 m newfile.txt new would split the file "newfile.txt" into three separate files called newaa, newab and newac each file the size of 22 MB.


13

Use a named pipe. mkfifo mysql_pipe gzip -9 -c < mysql_pipe > name_of_dump.gz & mysqldump database > mysql_pipe rm mysql_pipe I use it all the time, it'a awesome. http://en.wikipedia.org/wiki/Named_pipe


13

Use the mysqldump utility to direct the file wherever you like: mysqldump -A -u[username] -p[password] > /path/to/dest/backupname.sql If you need to, you can pipe the output through gzip: mysqldump -A -u[username] -p[password] | gzip -c > /path/to/dest/backupname.gz Further, you can send the output from gzip to another server via ssh: mysqldump ...


13

You probably want to buy more disk space, but assuming you don't, you could... pipe the tarball around rather than downloading it. newserver# ssh olduser@oldserver "cat /path/to/tarball" | tar xf - or if you don't have SSH access to your old server newserver# wget -O - http://oldserver/path/to/tarball | tar xf - or use rsync like Dennis said. Be ...


11

Rsync supports using ssh as a transport rsync -az /path/to/source username@host:/path/to/destination some older versions of rsync require you to specify ssh explicitly rsync -aze ssh /path/to/source host:/path/to/destination An alternative to using rsync is B. C. Pierce's Unison, which has similar functionality to rsync, but keeps a local index at both ...


11

If it compressed using gzip compression, you can do something like: gunzip -l filename.tar.gz Since a tar is not compressed, that should give you the information you need.


11

To strip down the parent directory: tar zxvf latest.tar.gz --strip 1 If you want to be sure that nothing from the parent directory is extracted: tar zxvf latest.tar.gz --strip 1 wordpress/directories Edit: --strip (--strip-path or --strip-components) was introduced in tar 1.14.


11

You can insert the pv command into your pipeline to get a report of how many bytes have been transferred, like this: tar -czpf - ./ --exclude mysql mysql-bin* mysql.sock | pv | ssh -lroot xxx.xx.xxx.xx tar -xzpf - -C /var/lib/mysql This will give you output like this, including the total number of bytes and the current transfer rate: 202MB 0:00:13 [ ...


11

You want to use the --strip-components=NUMBER option of tar: --strip-components=NUMBER strip NUMBER leading components from file names on extraction Your command would be: tar xfz /var/www/site.gz --strip-components=2 -C /tmp


10

If you just tar them up and nothing else this will waste tons of time with only minimal speed gain. So simply taring up the files with the cvf switches will effectively cost the time it takes to read all the 55GB images and write them back to disk. (Effectively it will be even more time wasted since there will be an considerable overhead). There is only ...


9

This small script seems to be your best option, given your requirements: cd directory for dir in */ do base=$(basename "$dir") tar -czf "${base}.tar.gz" "$dir" done It properly handles directories with spaces in their names.


9

Since it works over SSH, you can use: tar cjvf - /folder/to/backup | ssh root@ipv6 "cat > data.tar.bz2"


8

I wrote a quick script to suck down a remote mysql database. It uses mysql compression, gzip and ssh compression. Sucked down a multi GB database at an incredible rate. ssh -C user@host "mysqldump --opt --compress database <table> | gzip -9 -c" > outputfile.sql.gz A side benefit is that it requires no free space on the source database ...


8

Is your colleague a Windows guy? Because on Windows, there is the a (for archive) attribute which is set whenever a file is modified and is supposed to be cleared when a backup program backs it up. On Unix/Linux, backup programs normally use the ctime/mtime values of a file do decide if they need to backup the file - if the time is newer than in the ...


8

First create files with the filenames for each archive: find <directory> | split -l 5000 - files. Then create the tars: for f in files.*; do tar -cf $f.tar --files-from $f; done Untested but the basic idea should work.


8

tar records permissions based on the UID and GID, not on the string associated with them. So if the UID on one server was 3300 and that was linked to 'bob', on the new server the file will be owned by the user who has the UID 3300. Virtual everything (I want to say everything, but you can never be 100% sure) on UNIX uses the UID:GID values, because that's ...


7

ACK! several folks have suggested the wonderful date +%Y%m%d_%H%M%S style solution but nobody has mentioned the major caveat of '%' in crontabs... '%' it is equivalent to '\n' so your cronjob will likely fire and fail mystereously! You'll more likely want to simply escape it with backslash like this (and I also like to get some kind of inventory or other ...


7

tar (and cpio and afio and pax and similar programs) are stream-oriented formats - they are intended to be streamed direct to a tape or piped into another process. while, in theory, it would be possible to add an index at the end of the file/stream, i don't know of any version that does (it would be a useful enhancement though) it won't help with your ...


7

rsync is a quality tool to only copy deltas (changes in files): rsync --partial --progress --rsh=ssh ./public_html/ user@remotehost.com:/permitted/path This will backup the minimum amount of data possible to sync your two directories. In order to then create a backup, I'd recommend running the tar command in your question on the remote machine so you ...



Only top voted, non community-wiki answers of a minimum length are eligible