Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

For whatever reason, spammers found a way to relay mail through a side-system. The exploit has already been addressed.

The problem is that I had massive amount of emails in my /var/spool/mqueue directory (at least 100,000+) that I still need to filter through. I stopped sendmail and moved the contents of the mqueue directory to a new location...

Since then, I've been trying to use the following one liner to help:

for x in `find . -type f -name “qf*” | xargs grep -l "foo" | cut -b3-`; do y=d`echo $x | cut -b2-`; mv $x /root/spammessages; mv $y /root/spammessages/; done

The idea being to:

1) check QF files for unique spam-related header content (foo).

2) Find the DF counterpart file

3) Move both df and qf files to a isolated area.

The problem is that the query is running but doesn't look to be moving any files. If I run top, I see xargs and grep occasionally using some resources but never more than 1%-2%. Furthermore... when I check the spammessages folder, I don't see any files there.

If I run the same command on a smaller sub-set of messages, it seems to work fine. Are there some linux file limitations here? Are there ways to optimize the one-liner?

Thanks.

-M

share|improve this question

1 Answer 1

up vote 2 down vote accepted

Let's break it out into multiple lines

for x in `find . -type f -name "qf*" | xargs grep -l "foo" | cut -b3-`
do 
    y=d`echo $x | cut -b2-`
    mv $x /root/spammessages
    mv $y /root/spammessages/
done
  • You should always use -print0 with find and -0 with xargs when you are piping filenames from one to the other
  • Use $() instead of backticks for readability and the ability to nest and reduce the need for escapes
  • Always quote variable names that hold filenames
  • Calling an external (cut) many times in a loop is slow (I've eliminated it as it was not needed anyway)
  • You have a terminal slash on one directory but not the other (consistency)
  • You're cutting off the "qf" when you make the variable x, but you don't put it back when you do the move.
  • You're cutting off "qf" but not putting back "f" when you make the variable y (which you're cutting off an additional character from x)
  • Also, you had typographical quotes (smart or Unicode or whatever you want to call them) around your filespec in find which may have interfered

Try this:

for x in $(find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3-)
do 
    mv "qf$x" /root/spammessages/
    mv "df$x" /root/spammessages/
done

Put back into a one-liner:

for x in $(find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3-); do mv "qf$x" /root/spammessages/; mv "df$x" /root/spammessages/; done

Edit:

Here's a version using a while loop which may work better for very large numbers of files:

find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3- |
while read -r x
do 
    mv "qf$x" /root/spammessages/
    mv "df$x" /root/spammessages/
done

In a one-liner:

find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3- | while read -r x; do mv "qf$x" /root/spammessages/; mv "df$x" /root/spammessages/; done

A variation that uses Bash's process substitution:

while read -r x
do 
    mv "qf$x" /root/spammessages/
    mv "df$x" /root/spammessages/
done < <(find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3-)

And:

while read -r x; do mv "qf$x" /root/spammessages/; mv "df$x" /root/spammessages/; done < <(find . -type f -name "qf*" -print0 | xargs -0 grep -l "foo" | cut -b3-)
share|improve this answer
    
Thank you so very very much Dennis. I appreciate you taking the time to respond in detail. I have a couple quick follow-up questions: 1) The qf spammessages path is missing a / before root. I assume I should add one there? 2) Any thoughts on why my earlier command would work on smaller amounts of files but not large sets? –  Mike B Feb 8 '10 at 20:06
1  
@Mikey B: Yeah, that missing slash is a typo. I'll fix it in my answer. I can't come up with a reason that the original would work at all. You didn't get any error messages on the larger file set? On the small set were all files moved that you expected? Are there any differences between the filenames in the smaller set and the larger one? I'm going to add another version in my answer that will probably handle larger number of files in case you're overrunning your command length limit using a for statement. –  Dennis Williamson Feb 8 '10 at 21:35
    
Brilliant! Thank you sir! –  Mike B Feb 8 '10 at 22:15
    
I think the one-liner in the middle should actually have cut -b5- instead of cut -b3- –  Mike B Aug 8 '12 at 21:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.