Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a 30gb disk image of a borked partition (think dd if=/dev/sda1 of=diskimage) that I need to recover some text files from. Data carving tools like foremost only work on files with well defined headers, i.e. not plain text files, so I've fallen back on my good friend strings.

strings diskimage > diskstrings.txt produced a 3gb text file containing a bunch of strings, mostly useless stuff, mixed in with the text that I actually want.

Most of the cruft tends to be really long, unbroken strings of gibberish. The stuff I'm interested in is guaranteed to be less than 16kb, so I'm going to filter the file by line length. Here's the Python script I'm using to do so:

infile  = open ("infile.txt" ,"r");
outfile = open ("outfile.txt","w");
for line in infile:
    if len(line) < 16384:
        outfile.write(line)
infile.close()
outfile.close()

This works, but for future reference: Are there any magical one-line incantations (think awk, sed) that would filter a file by line length?

share|improve this question
add comment

4 Answers 4

up vote 12 down vote accepted
awk '{ if (length($0) < 16384) print }' yourfile >your_output_file.txt

would print lines shorter than 16 kilobytes, as in your own example.

Or if you fancy Perl:

perl -nle 'if (length($_) < 16384) { print }' yourfile >your_output_file.txt
share|improve this answer
    
Well, that was embarrasingly simple. Thank you. :) –  Li-aung Yip Jan 31 '12 at 8:32
    
Added also Perl version :-) –  Janne Pikkarainen Jan 31 '12 at 8:38
    
And the awk script can be written as awk 'length($0) < 16384' file > output, as the default action is to print the line. –  glenn jackman Jan 31 '12 at 16:19
add comment

This is similar to Ansgar's answer, but slightly faster in my tests:

awk 'length($0) < 16384' infile >outfile

It's the same speed as the other awk answers. It relies on the implicit print of a true expression, but doesn't need to take the time to split the line as Ansgar's does.

Note that AWK gives you an if for free. The command above is equivalent to:

awk 'length($0) < 16384 {print}' infile >outfile

There's no explicit if (or its surrounding set of curly braces) as in some of the other answers.

Here is a way to do it in sed:

sed '/.\{16384\}/d' infile >outfile

or:

sed -r '/.{16384}/d' infile >outfile

which delete any line that contains 16384 (or more) characters.

For completeness, here's how you'd use sed to save lines longer than your threshold:

sed '/^.\{0,16383\}$/d' infile >outfile
share|improve this answer
add comment

Not really different from the answers already given, but shorter still:

awk -F '' 'NF < 16384' infile >outfile
share|improve this answer
add comment

You can awk such as:

$ awk '{ if (length($0) < 16384) { print } }' /path/to/text/file

This will print the lines longer shorter than 16K characters (16 * 1024).

You can use grep also:

$ grep ".\{,16384\}" /path/to/text/file

This will print the lines at most 16K characters.

share|improve this answer
    
Not sure grep is such a good idea - it's a simple regexp, to be sure, but more computationally expensive than awk. "A man with problem says "I'll use regular expressions!" Now he has two problems." ;) –  Li-aung Yip Jan 31 '12 at 8:54
    
It is just another way of doing it. The first option I posted was using awk. –  Khaled Jan 31 '12 at 8:58
    
+1 for the regexp, because it golfs better, and it does not make me read awk manpages =) –  Ciro Santilli Mar 12 at 21:29
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.