Find files containing a string on the whole filesystem

Question

I need to find all the instances of a given string in the whole filesystem, because I don't remember in which configuration files, script or any other programs I put it and I need to update that string with a new one.

I tried with the following command

`grep -nr 'needle' / --exclude-dir=.svn | mail [email protected] -s 'References on xxx'

If I run this command on a small directory it gives me the output I need in the form

/path1/:nn:line containing needle
/path2/:nn:line containing needle

where /path1 is the full path of the file, nn is the row containing the needle and last field is the content of the line.

However when I run the command on the root directory the grep process hang after a while. I run this script about 8 hours ago and even on a small filesystem (less than 5GB) it doesn't end and if I run top or ps the process seems sleeping

root 24909 0.0 0.1 3772 1520 pts/1 S+ Feb10 0:15 grep -nr needle / --exclude-dir=.svn

Why it doesn't end? Is there any better way to do this (it's a one time job, I don't need to execute this more than once)

Thanks.

Update: I found a working solution with find and xargs which seems to be working and uses less system resources than the find -exec solution. Here is my final command line:

find /{boot,etc,home,lib,lost+found,opt,root,sbin,usr,var} -type f -print0 | xargs -r0 grep -nr 'needle' | mail [email protected] -s 'References on xxx'

I used the /{dir1,dir2,...,dirn} syntax because I want the output lines with the fullpath including the leading slash, and in this way you can use the command without cd / i.e. from any directory.

Answer 1

There are some files in the file system that aren't real files, but are instead hooks into the kernel. Some of those can be read from forever. Try

grep foo /dev/zero

to see this in action. Get ready to stop it with ctrl-C before it takes over the whole system.

If I wanted to do what you're doing, I'd enumerate the subdirectories of / that I wanted scanned, and make sure I only checked text files, probably with

cd /
find boot etc home lib lost+found media mnt opt root sbin tmp usr var -type f -exec grep needle {} /dev/null \;

Note how the list does not contain /dev, /proc, /sys or /selinux.

Answer 2

You can try using find + xargs + grep for that, kinda:

find /there -type f :MaybeSomeRestrictingFlagsLikeSizeNotBigger500MB_or_FS_type_if_u_know_its_exactly_on_EXT3_AndSoOn: -print0 | xargs -0r grep needle /dev/null

(/dev/null makes grep printing file name even if only one file was found)

Answer 3

I don't know why it doesn't end but "fgrep -R" usually works in those cases.

Answer 4

You might want to exclude dev and proc from that command. I would do it with find instead and exclude special files:

find . -type f -exec grep -H "My search string" '{}' \; -print 

Answer 5

As some of the others pointed out, the error is occuring because you are using grep on some 'files' that will produce weird results. For example grep XXX /dev/zero will go on forever.

I would first exclude the directories such as /dev/ and /proc/ from your search. Another way is to only search for 'files' and no other types.

Also if your argument list in grep is too long, you will get this error:

bash: /bin/grep: Argument list too long

You can get around this with a loop, such as this:

for i in `find -type f /`; do grep -H "sample string" $i; done

Or by executing it from find, whatever floats your boat.

find / -type f -exec grep -H "sample string" '{}' \; -print 

This should do a find and then grep each file. Although some have pointed out that it is inefficient to spawn grep each time you find a file, you will have to do something which will be 'inefficient' because you do literally have to search every file for that string.