Tag Info

Hot answers tagged

60

mysqldump --opt <database> | gzip -c | ssh user@wherever 'cat > /tmp/yourfile.sql.gz' You can't use tar in a pipe like this, and you don't need it anyway, as you're only outputting a single file. tar is only useful if you have multiple files.


52

tar has 3 types of syntax (according to this ): long options (--file) short options (-f) old options (f) For the old option syntax, all the letters must follow "tar" and must all fall into a single clump with no spaces. The order of the letters doesn't really matter, so long as the arguments to those letters follow the same order after the clump of ...


49

I'd be tempted to rsync it over myself - it does compression and handles link loss well.


30

Instead of using tar to write to your local disk, you can write directly to the remote server over the network using ssh. server1$ tar -zc ./path | ssh server2 "cat > ~/file.tar.gz" Any string that follows your "ssh" command will be run on the remote server instead of the interactive logon. You can pipe input/output to and from those remote commands ...


25

Putting every file into a separate tar file doesn't make any sense in this scenario. You can use gzip to compress them directly: gzip * will result in file1.out.gz, file2.out.gz etc. You would use tar only if you would need a compressed archive as a single file. If you ineed need a tar archive for every file, you can create it like so: for i in *; ...


21

Don't tar. Use rsync -av to preserve permissions while transfering the files. Though like tar, this does not preserve selinux context. Not that I would consider that important though.


17

pigz is a parallel version of gzip. Although it only uses a single thread for decompression, it starts 3 additional threads for reading, writing, and check calculation. Your results may vary but we have seen significant improvement in decompression of some of our data sets. Once you install pigz the tar file can be extracted with: pigz -dc target.tar.gz ...


15

What you're looking for is the -C option: $ man tar <snip> -C, --directory DIR change to directory DIR Usage, per your example: $ tar xvzf archivename.tar.gz -C /tmp/testdir/ sampledir/ Also, you may be interested to know that modern versions of tar don't require the z flag to decompress gzipped files; they'll notice it's a .gz and Do The ...


15

If you install version 3+ of rsync it will do a rolling list of files to transfer and won't need to keep the entire file list in memory. In the future you probably want to consider hashing the filenames and creating a directory structure based on parts of those hashes. You can see this answer to get an idea of what I mean with the hashing.


14

Use the split command. split -b 22 m newfile.txt new would split the file "newfile.txt" into three separate files called newaa, newab and newac each file the size of 22 MB.


14

I don't know if this one is "the best", but you can try 7-Zip


14

Summing up previous answers and adding some important information: When creating archives, tar will always preserve files' user and group ID, unless told otherwise with --owner=NAME, --group=NAME. But still there will always be a user and group associated with each file. GNU tar, and perhaps other versions of tar, also store the user and group names, ...


13

Use a named pipe. mkfifo mysql_pipe gzip -9 -c < mysql_pipe > name_of_dump.gz & mysqldump database > mysql_pipe rm mysql_pipe I use it all the time, it'a awesome. http://en.wikipedia.org/wiki/Named_pipe


13

You probably want to buy more disk space, but assuming you don't, you could... pipe the tarball around rather than downloading it. newserver# ssh olduser@oldserver "cat /path/to/tarball" | tar xf - or if you don't have SSH access to your old server newserver# wget -O - http://oldserver/path/to/tarball | tar xf - or use rsync like Dennis said. Be ...


13

Use the mysqldump utility to direct the file wherever you like: mysqldump -A -u[username] -p[password] > /path/to/dest/backupname.sql If you need to, you can pipe the output through gzip: mysqldump -A -u[username] -p[password] | gzip -c > /path/to/dest/backupname.gz Further, you can send the output from gzip to another server via ssh: mysqldump ...


11

If it compressed using gzip compression, you can do something like: gunzip -l filename.tar.gz Since a tar is not compressed, that should give you the information you need.


11

Rsync supports using ssh as a transport rsync -az /path/to/source username@host:/path/to/destination some older versions of rsync require you to specify ssh explicitly rsync -aze ssh /path/to/source host:/path/to/destination An alternative to using rsync is B. C. Pierce's Unison, which has similar functionality to rsync, but keeps a local index at both ...


11

You can insert the pv command into your pipeline to get a report of how many bytes have been transferred, like this: tar -czpf - ./ --exclude mysql mysql-bin* mysql.sock | pv | ssh -lroot xxx.xx.xxx.xx tar -xzpf - -C /var/lib/mysql This will give you output like this, including the total number of bytes and the current transfer rate: 202MB 0:00:13 [ ...


10

If you just tar them up and nothing else this will waste tons of time with only minimal speed gain. So simply taring up the files with the cvf switches will effectively cost the time it takes to read all the 55GB images and write them back to disk. (Effectively it will be even more time wasted since there will be an considerable overhead). There is only ...


10

To strip down the parent directory: tar zxvf latest.tar.gz --strip 1 If you want to be sure that nothing from the parent directory is extracted: tar zxvf latest.tar.gz --strip 1 wordpress/directories Edit: --strip (--strip-path or --strip-components) was introduced in tar 1.14.


9

Since it works over SSH, you can use: tar cjvf - /folder/to/backup | ssh root@ipv6 "cat > data.tar.bz2"


8

First create files with the filenames for each archive: find <directory> | split -l 5000 - files. Then create the tars: for f in files.*; do tar -cf $f.tar --files-from $f; done Untested but the basic idea should work.


8

I wrote a quick script to suck down a remote mysql database. It uses mysql compression, gzip and ssh compression. Sucked down a multi GB database at an incredible rate. ssh -C user@host "mysqldump --opt --compress database <table> | gzip -9 -c" > outputfile.sql.gz A side benefit is that it requires no free space on the source database ...


8

This small script seems to be your best option, given your requirements: cd directory for dir in */ do base=$(basename "$dir") tar -czf "${base}.tar.gz" "$dir" done It properly handles directories with spaces in their names.


8

Is your colleague a Windows guy? Because on Windows, there is the a (for archive) attribute which is set whenever a file is modified and is supposed to be cleared when a backup program backs it up. On Unix/Linux, backup programs normally use the ctime/mtime values of a file do decide if they need to backup the file - if the time is newer than in the ...


7

rsync is a quality tool to only copy deltas (changes in files): rsync --partial --progress --rsh=ssh ./public_html/ user@remotehost.com:/permitted/path This will backup the minimum amount of data possible to sync your two directories. In order to then create a backup, I'd recommend running the tar command in your question on the remote machine so you ...


7

ACK! several folks have suggested the wonderful date +%Y%m%d_%H%M%S style solution but nobody has mentioned the major caveat of '%' in crontabs... '%' it is equivalent to '\n' so your cronjob will likely fire and fail mystereously! You'll more likely want to simply escape it with backslash like this (and I also like to get some kind of inventory or other ...


7

tar (and cpio and afio and pax and similar programs) are stream-oriented formats - they are intended to be streamed direct to a tape or piped into another process. while, in theory, it would be possible to add an index at the end of the file/stream, i don't know of any version that does (it would be a useful enhancement though) it won't help with your ...


7

Any reason for tar: Can't specify both -x and -t?* Yes: Because in the designer's eyes this would be useless. If you have a tar file you either want to extract its contents (-x), or you want to list them (-t). Much like you can't specify -c -t (You're creating an archive, why would you want to list the things you're adding to it, which you've presumably ...


7

Your best bet is to use lsof to determine if a file has been opened by any process: # lsof -f -- /var/log/syslog COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 1520 syslog 1w REG 252,2 72692 16719 /var/log/syslog You can't easily tell if it's in the process of being written to, but if it is being written to, it MUST be ...



Only top voted, non community-wiki answers of a minimum length are eligible