Finding matches from one large file in another?

Question

I have two email lists. One is a newline delimited file of just email addresses containing 100k lines and the second file contains email,date,ipaddress newline. It has 4M lines, and contains duplicates which I am not concerned with.

grep -f fileA.txt fileB.txt works when fileA.txt is a test file of 100 or 1000 lines but 100k it is isn't getting anywhere.

I'm open to perl as well :)

Answer 1

When faced with this kind of thing and I don't/can't store all of one file in an array, as suggested by Eric, I resort to a slightly unconventional approach. Each file is exported to a separate table in a database (I like Perl for this part) and the desired results are obtained via SQL queries.

Answer 2

You may be able to speed it up a bit by using the -F option so it's searching for fixed strings.

grep -Ff fileA.txt fileB.txt

Did you time your tests? What does extrapolating that time to the larger files tell you?

Answer 3

Sort and then diff them? That ought to work.

Answer 4

I'm assuming Linux with this. I would try creating a ramdisk and putting both files in it. That may be the fastest thing to try. Put this in fstab and then do a mount /mnt for a really quick way of setting it up:

ramdisk /mnt tmpfs mode=1777,size=1G

Answer 5

in perl:

#!/usr/bin/perl -w

my ($emailfile, $file2);

if ( open ( $emailfile, '/path/file') )
{
  my @emails = split(/\n/, $emailfile);

  if ( open ( $file1, '/path/file') )
  {
    foreach my $email ( @emails )
    {
      while ( <$file2> )
      {
        if ( $_ =~ /$email/)
          print $_;
      }
    }
  }
}