Linux / Sendmail one liner having trouble with large directories

Question

For whatever reason, spammers found a way to relay mail through a side-system. The exploit has already been addressed.

The problem is that I had massive amount of emails in my /var/spool/mqueue directory (at least 100,000+) that I still need to filter through. I stopped sendmail and moved the contents of the mqueue directory to a new location...

Since then, I've been trying to use the following one liner to help:

for x in `find . -type f -name “qf*” | xargs grep -l "foo" | cut -b3-`; do y=d`echo $x | cut -b2-`; mv $x /root/spammessages; mv $y /root/spammessages/; done

The idea being to:

1) check QF files for unique spam-related header content (foo).

2) Find the DF counterpart file

3) Move both df and qf files to a isolated area.

The problem is that the query is running but doesn't look to be moving any files. If I run top, I see xargs and grep occasionally using some resources but never more than 1%-2%. Furthermore... when I check the spammessages folder, I don't see any files there.

If I run the same command on a smaller sub-set of messages, it seems to work fine. Are there some linux file limitations here? Are there ways to optimize the one-liner?

Thanks.

-M

Answer 1

Let's break it out into multiple lines

for x in `find . -type f -name "qf*" | xargs grep -l "foo" | cut -b3-`
do 
    y=d`echo $x | cut -b2-`
    mv $x /root/spammessages
    mv $y /root/spammessages/
done

• You should always use -print0 with find and -0 with xargs when you are piping filenames from one to the other
• Use $() instead of backticks for readability and the ability to nest and reduce the need for escapes
• Always quote variable names that hold filenames
• Calling an external (cut) many times in a loop is slow (I've eliminated it as it was not needed anyway)
• You have a terminal slash on one directory but not the other (consistency)
• You're cutting off the "qf" when you make the variable x, but you don't put it back when you do the move.
• You're cutting off "qf" but not putting back "f" when you make the variable y (which you're cutting off an additional character from x)
• Also, you had typographical quotes (smart or Unicode or whatever you want to call them) around your filespec in find which may have interfered

Try this:

for x in $(find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3-)
do 
    mv "qf$x" /root/spammessages/
    mv "df$x" /root/spammessages/
done

Put back into a one-liner:

for x in $(find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3-); do mv "qf$x" /root/spammessages/; mv "df$x" /root/spammessages/; done

Edit:

Here's a version using a while loop which may work better for very large numbers of files:

find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3- |
while read -r x
do 
    mv "qf$x" /root/spammessages/
    mv "df$x" /root/spammessages/
done

In a one-liner:

find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3- | while read -r x; do mv "qf$x" /root/spammessages/; mv "df$x" /root/spammessages/; done

A variation that uses Bash's process substitution:

while read -r x
do 
    mv "qf$x" /root/spammessages/
    mv "df$x" /root/spammessages/
done < <(find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3-)

And:

while read -r x; do mv "qf$x" /root/spammessages/; mv "df$x" /root/spammessages/; done < <(find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3-)