Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have two email lists. One is a newline delimited file of just email addresses containing 100k lines and the second file contains email,date,ipaddress newline. It has 4M lines, and contains duplicates which I am not concerned with.

grep -f fileA.txt fileB.txt works when fileA.txt is a test file of 100 or 1000 lines but 100k it is isn't getting anywhere.

I'm open to perl as well :)

share|improve this question

5 Answers 5

When faced with this kind of thing and I don't/can't store all of one file in an array, as suggested by Eric, I resort to a slightly unconventional approach. Each file is exported to a separate table in a database (I like Perl for this part) and the desired results are obtained via SQL queries.

share|improve this answer

in perl:

#!/usr/bin/perl -w

my ($emailfile, $file2);

if ( open ( $emailfile, '/path/file') )
{
  my @emails = split(/\n/, $emailfile);

  if ( open ( $file1, '/path/file') )
  {
    foreach my $email ( @emails )
    {
      while ( <$file2> )
      {
        if ( $_ =~ /$email/)
          print $_;
      }
    }
  }
}
share|improve this answer
    
Won't that try to put all file in the memory, for each one of the loops? –  coredump Feb 18 '11 at 1:46
    
true, sorry, you can also place the foreach loop inside the while loop to only iterate the larger file once. I just went a little too fast for myself :/ –  Eric Fossum Feb 18 '11 at 1:53

I'm assuming Linux with this. I would try creating a ramdisk and putting both files in it. That may be the fastest thing to try. Put this in fstab and then do a mount /mnt for a really quick way of setting it up:

ramdisk /mnt tmpfs mode=1777,size=1G
share|improve this answer
    
I don't think speed is the issue. The OP seems to be asking how to do this and with only 100k lines that should occur in the blink of an eye. –  John Gardeniers Feb 18 '11 at 2:00

Sort and then diff them? That ought to work.

share|improve this answer
    
The files are different sizes, diff wont work I think –  coredump Feb 18 '11 at 1:45
    
Diff will work on files of different sizes. I do like John's answer of using SQL, that's something that it's very good at. –  mfinni Feb 18 '11 at 14:37

You may be able to speed it up a bit by using the -F option so it's searching for fixed strings.

grep -Ff fileA.txt fileB.txt

Did you time your tests? What does extrapolating that time to the larger files tell you?

share|improve this answer
    
A test I just performed showed a considerable difference when using -F. time grep -Fqsf 30000-lines 900000-lines took about 1.2% of the time of time grep -qsf 10000-lines 900000-lines! Note that's with three times as many lines in the pattern file! –  Dennis Williamson Feb 18 '11 at 1:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.