Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I want to remove all lines where the second column is 05408736032.

0009300|05408736032|89|01|001|0|0|0|1|NNNNNNYNNNNNNNNN|asdf|
0009367|05408736032|89|01|001|0|0|0|1|NNNNNNYNNNNNNNNN|adff|
share|improve this question
    
Double posted: stackoverflow.com/questions/1439816/… –  Dennis Williamson Sep 17 '09 at 19:19
add comment

5 Answers 5

up vote 3 down vote accepted

This might do what you want:

sort -t '|' -k 2,2 -u  foo.dat

However this sorts the input according to your field, which you may not want. If you really only want to remove duplicates, your best option is Perl:

perl -ne '$a=(split "\\|")[1]; next if $h{$a}++; print;' foo.dat
share|improve this answer
add comment
awk -F \| '{if ($2 != 05408736032) print}'
share|improve this answer
1  
You can leave out the "if" and the "print": awk -F \| '$2 != "05408736032"' –  Dennis Williamson Sep 17 '09 at 18:24
add comment

Pure Bash:

oldIFS=$IFS
while read line
do
    IFS=$'|'
    testline=($line)  # make an array split according to $IFS
    IFS=$oldIFS       # put it back as soon as you can or you'll be sooOOoorry
    if [[ ${testline[1]} != "05408736032" ]]
    then
        echo $line
    fi
done < datafile
share|improve this answer
add comment

You can do something like:

for f in `cat $file`; do 
  val=`echo $f | cut -d\| -f 2`
  if [ `grep $val $file | wc -l` -lt 2 ]; then
     echo $f
  fi
done

but, like most shell scripts, it's pretty inefficient. You'd be better off doing it in perl, something like:

@infile=<>;

foreach (@infile) {

  @foo = split(/|/);
  if exists $found{$foo[1]} {
    $found{$foo[1]}++;
  } else {
    $found{$foo[1]}++;
  }

}

foreach (@infile) {
  @foo = split(/|/);
  if ($found{$foo[1]} < 2) {
    print $_;
  }
}
share|improve this answer
add comment

Is it that you want to remove all lines where the second | separated field contains '05408736032'? Will all the lines be formatted the same? If so, this should output the file minus those lines (it's perl that takes the original file as the first argument and the file it's going to as the second).

#!/usr/bin/perl
use warnings;
use strict;
my  ($file1, $file2) = @ARGV;
open my $origin_file, '<', $file1;
open my $newfile, '>', $file2;
while (my $line = <$origin_file>) {
    my @values = split '/|/', $line;
    print $newfile $line unless $vaules[1] = '05408736032';
}
close $newfile or die $!;
close $origin_file or die $!;

(I haven't tested this, so you probably want to backup the original file before you try it)

On reading again, you may be looking to grab only lines with a unique second column. This should do that.

#!/usr/bin/perl
use warnings;
use strict;
my  ($file1, $file2) = @ARGV;
open my $origin_file, '<', $file1;
open my $newfile, '>', $file2;
while (my $line = <$origin_file>) {
    my @values = split '/|/', $line;
    print $newfile $line unless defined $unique{$values[1]};
    $unique{$vaules[1]} += 1;
}
close $newfile or die $!;
close $origin_file or die $!;
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.