Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Given a base directory (like /home/user) is there a command that could be run which would create an archive back up all text files (i.e. files less than 100kb) in that directory and it's children? I know you can tar/gz a directory - but what about excluding files also?

The idea is that most photos, videos, and other large files would be ignored while all important hand-typed documents could easily be backed up quickly when moving around projects and servers.

UPDATE

Using skinp's awesome code I was able to backup a small amount of the files. However, as DerfK pointed out - there is a limit to the size of arguments you can pass to commands. With that in mind I was able to write out the contents of find hoping that I could use something to read the contents to tar and bypass this limit. The other other option seems to be a sh script that could ad each file to the archive at a time.

find /home/username -type f -size -100k > list.of.small.files.txt

Which rendered out a 6MB file.

share|improve this question
1  
too tired to build a whole solution: have a look at tar, find and xargs. –  ring0 Oct 19 '10 at 18:12

3 Answers 3

up vote 4 down vote accepted

Looking at the other answers posted here so far, I'm concerned that the uses I see of xargs and find -exec {} are erroneous. If and when the file list grows long enough that tar -c is executed by xargs more than once, the tar file populated to that point will be overwritten. Thus, only the files from the last invokation of tar will end up in the tarfile.

Here's a one-liner that should always work, regardless of the total number of files, and regardless of whether filenames contain embedded newlines:

find /home/user -type f -size -100k -print0 | tar -c -z --null --files-from=- -f backup.tgz

The find command generates a list of null-terminated file names to backup, and the tar command reads that list from the pipe and creates the tarfile backup.tgz.

share|improve this answer
    
Awesome, next thing I knew I had a 100MB backup of text files sitting there. –  Xeoncross Oct 20 '10 at 17:30

I would use the power of find:

find /home/user -type f -size -100k -exec tar cvzf backup.tgz {} \+

type: specifies you want a file not a directory
size: with the number preceeded by -, means we want less than 100k
exec: execute the tar with {} being the files found, \+ means end of the exec

You could also use xargs:

find /home/user -type f -size -100k | xargs tar cvzf backup.tgz

Update:

tar as a command to append a file to an existing tar archive (maybe even not existing, it works for me).

Here's a simple example script doing this:

find . -type f -size -100k -print > filelist
for i in `cat filelist`
do
    tar --append $i --file=backup.tar
done
gzip backup.tar

Obviously, this script is highly ineficient... It only append one file at a time, launching the tar command as many time as there is files.
It would be good to script it in a way that it append say 1000 files each pass...

share|improve this answer
    
You will have a lot of tars :-) –  ring0 Oct 19 '10 at 18:25
1  
From what I remember, \+ means execute the command once for all that was found, \; means execute the command each time something is found. I tried it and it worked... could be system dependent I guess though. –  skinp Oct 19 '10 at 18:31
    
Both work, (using lain's modifications) so which would be better/faster on large directories? –  Xeoncross Oct 19 '10 at 18:42
1  
@Xeoncross both options are essentially equivalent, but the xargs option requires starting a third program (xargs) so could be a few ms slower. Also, by "large" do you mean really large? Be careful, if you have more than 131k or so files you'll hit the ARG_MAX limit on commandline arguments with either the xargs or the find command and you'll need a different option for getting them all in the tarball. See in-ulm.de/~mascheck/various/argmax/#results –  DerfK Oct 19 '10 at 19:01
1  
@Xeoncross have you used tar tzf backup.tgz (possibly add | wc to get a line count rather than looking file by file) to confirm that you didn't get all the files? Text tends to compress really well. You could also run find /home/user -type f -size -100k | less and look through it to see if you're finding the files you thought you'd get (or wc instead of less to compare the line count to the tar command above) –  DerfK Oct 19 '10 at 19:58

Try

find /home/user -type f -size -100k -print0 | xargs -0 tar cvzf tarfile.tar.gz

which will be safe for files with spaces in the name too.

share|improve this answer
    
Danger! This will fail if/when the list of files grows longer than the maximum number of command-line arguments. xargs executes the given command more than once if it can't fit all the arguments into one command invokation. In this case, each time xargs runs another copy of tar -c, the tarfile will be overwritten, and the final result will contain only the last batch of filenames that xargs received. You really need to do a tar -A instead. –  Steven Monday Oct 20 '10 at 1:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.