Hot answers tagged

63

tar has 3 types of syntax (according to this ): long options (--file) short options (-f) old options (f) For the old option syntax, all the letters must follow "tar" and must all fall into a single clump with no spaces. The order of the letters doesn't really matter, so long as the arguments to those letters follow the same order after the clump of options....


55

Summing up previous answers and adding some important information: When creating archives, tar will always preserve files' user and group ID, unless told otherwise with --owner=NAME, --group=NAME. But still there will always be a user and group associated with each file. GNU tar, and perhaps other versions of tar, also store the user and group names, unless ...


27

You can create an SSH tunnel through machine2 then in another session connect to the tunnel. For example, open two CLI sessions on machine1. In the first session run the following: MACHINE1$ ssh -L 2022:MACHINE3:22 <user>@MACHINE2 In the second session run the following: MACHINE1 $ ssh -p 2022 <user>@localhost What's happening with the ...


22

With gnu tar, you can create an archive with a different base directory than the actual with: tar -c --transform 's,^\.,mybasedir,' . Just adapt the sed expression to your needs.


21

Don't tar. Use rsync -av to preserve permissions while transfering the files. Though like tar, this does not preserve selinux context. Not that I would consider that important though.


18

In your case it is the file level encryption that is preventing compression. Encryption tries to make the data stream look as much as random "noise" as possible. Compression tries to increase the data "density" which has a similar effect of limiting further compression.


16

The fact is tar is successful, but prints out warning that those socket files are ignored GNU tar actually provides an option to suppress warnings. You could ignore the "socket ignored" errors using the below command which avoids running a find command that could take a long time to complete. tar --warning='no-file-ignored' -cpzf backup_name.tar.gz /...


15

Your best bet is to use lsof to determine if a file has been opened by any process: # lsof -f -- /var/log/syslog COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 1520 syslog 1w REG 252,2 72692 16719 /var/log/syslog You can't easily tell if it's in the process of being written to, but if it is being written to, it MUST be open. ...


15

GNU tar has a -C option for this. -C, --directory=DIR change to directory DIR So you could do something like: tar -C /var/www/website -zcvf /tmp/mytar.tar.gz . Untarring (you will do that eventually) is the same: tar -C /var/www/website -zxvf /tmp/mytar.tar.gz


14

Note that your tar command is completing successfully; it's just complaining about the socket entries. Tiffany is suggesting a mechanism for filtering out those particular error messages, although tar will still exit with an error code. You could also feed a list of sockets to tar's -X option to have them excluded from the backup, e.g: # find / -type s -...


13

You are on the right track, renaming the file is an atomic operation, so performing the rename after upload is simple, elegant and not error prone. Another approach I can think of is to use lsof | grep filename.tar.gz to check if the file is being accessed by another process.


13

Use the mysqldump utility to direct the file wherever you like: mysqldump -A -u[username] -p[password] > /path/to/dest/backupname.sql If you need to, you can pipe the output through gzip: mysqldump -A -u[username] -p[password] | gzip -c > /path/to/dest/backupname.gz Further, you can send the output from gzip to another server via ssh: mysqldump -A ...


12

I think this has been answered here: There is the nocache utility, which can prepended to a command like ionice and nice. It works by preloading a library which adds posix_fadvise with the POSIX_FADV_DONTNEED flag to any open calls. In simple terms, it advises the kernel that caching is not needed for that particular file; the kernel will then ...


12

You probably want to buy more disk space, but assuming you don't, you could... pipe the tarball around rather than downloading it. newserver# ssh olduser@oldserver "cat /path/to/tarball" | tar xf - or if you don't have SSH access to your old server newserver# wget -O - http://oldserver/path/to/tarball | tar xf - or use rsync like Dennis said. Be ...


11

You could use SquashFS for such archives. It is designed to be accessed using a fuse driver (although a traditional interface exists) compressed (the larger the block size, the more efficient) included in the Linux kernel stores UIDs/GIDs and creation time endianess-aware, therefore quite portable The only drawback I know of is that it is read-only. http:/...


9

Since it works over SSH, you can use: tar cjvf - /folder/to/backup | ssh root@ipv6 "cat > data.tar.bz2"


9

tar records permissions based on the UID and GID, not on the string associated with them. So if the UID on one server was 3300 and that was linked to 'bob', on the new server the file will be owned by the user who has the UID 3300. Virtual everything (I want to say everything, but you can never be 100% sure) on UNIX uses the UID:GID values, because that's ...


9

Try the --xform option, which uses a sed-like replace expression. This allows you to change the path references going into the archive rather than handle it during the extraction (such as with the --strip-components). tar --exclude=.gitignore -cz --xform s:'./':: -f dist.tgz ./ To list the files as they are processed, add the -v and --show-transformed-name ...


9

Sockets are zero level files that are used by daemon processes to communicate with each other. They are created and destroyed as necessary when the daemons start and stop. They can safely be ignored. You can always get rid of them with tar <my_options> 2> >(grep -v 'socket ignored' >&2)


8

In case anyone else comes across this (as I just have) there are some reasonably detailed (and mostly correct) steps over here. Key details The key point is to unpack all of the duplicity-full.*.difftar.gz files in the same place, so that you're left with just two snapshot/ and multivol_snapshot/ directories. If your file is in snapshot/ then you're done. ...


7

Is there some parameter that allows my tar file to be resumed after some problem? Nope. What you should do, though, is run your command from within a terminal multiplexer like screen or tmux. That way if your connection drops, the process keeps running.


6

To make a tar file with a consistent checksum, just prepend GZIP=-n like this: GZIP=-n tar -zcf myOutputTarball.tar /home/luke/directoryIWantToZip How this works: Tar can accept gzip options using a temporary GZIP environment variable, as above. Like Valter said, tar uses gzip, which by default puts a timestamp in the archive. This means you get a ...


6

If you want to see progress use something like pv. Here is an Example: pigz -dc mysql-binary-backup.tar.gz | pv | tar xf -


6

A bit old, but most of the answers completely misses the point of the question: But I figured I'd try to figure out if there is simply a way to determine if the file is whole at the command line first... In general, there isn't. You simply don't have enough information to determine that. Because determining that the file is closed is not the same as ...


6

It's possible some blocks from the tars will be the same, but very unlikely. ZFS deduplicates at the block level (called the recordsize in ZFS parlance), so individual blocks need to be identical. The tars are essentially guaranteed to have runs of the exact same content, but whether that will compose a whole block and be block aligned is highly unlikely. ...


6

I think I would actually do this using find and then pass that input into tar. Using your example, let's assume you want files that are between 60 and 90s days old. find /home/public_html/images -type f -daystart -mtime -90 -and -mtime +60 -print0 | xargs -0 tar -Ajf images_60-90.tar.bz2 This will list all the files that were last modified more than 60 ...


6

You're compressing (gzip, even) 62 GB of data. At 40 MB/s, that transfer would take ~30 minutes on its own. However, gzip involves the CPU greatly. If you're pinning both cores at 100%, then there's your answer.


6

By default tar does not provide a means of skipping empty directories. That being said, it is not too much work to pull a simple shell-script out of thin air to do what needs to be done. Here is a forum discussion where I found the solution to the exact same problem, when the need arose. MAJOR UPDATE: First I need to seriously apologize for my sloppy copy-...


6

Your assumption is correct, "File changed as we read it" is a notice, usually related to files in use (i.e. written to during the creation process) while tar is creating the archive. If consistency is vital, you're better off rsyncing the contents elsewhere i.e. rsync -avz /my/home/ /somebackupdir/my/home/ # initial sync, followed by rsync -avz /my/...


6

Use the appropriate tar command line option: --one-file-system stay in local file system when creating archive


Only top voted, non community-wiki answers of a minimum length are eligible