8

I need to run a grep on my server to search for files which extend a CodeIgniter 1 file, as we are upgrading to CI2. In one folder contains several hundred site-specific folders like such:

dev/sitea.com/site/www
dev/siteb.com/site/www
dev/sitec.com/site/www
... and so on

How can I grep to find which folders directly under dev contain a file (or many files in variable subfolders) which references the string "MY_Output" (php file)

I'm aware of the -l flag to list the files, but I'm not sure how to put it with other flags together to say "show me which websites contain files with this string"

Is this possible? Thanks!

EDIT: Just to clarify, these site level folders will contain files which extend MY_Output.php, which is a common shared file:

class Whatever extends MY_Output
5
  • This is not related to apache, ssh or putty (thus I removed the tags). Jul 15, 2011 at 18:42
  • I'm using Putty to run these greps. Figured it may be applicable.
    – AlienWebguy
    Jul 15, 2011 at 19:42
  • 2
    Maybe bash could be applicable (as this is most likely the shell you are executing the command in), but the command should not depend on how you connect. (Please do not regard this as an attack, I just wanted to mention why I removed the tags, in case anyone wonders.) Jul 15, 2011 at 19:45
  • Perhaps you could include something about what a positive match might look like. For instance would the file /dev/sitea.com/site/www/file contain the text "MY_Output" within it, or are you looking for "MY_Output" within the path itself?
    – Red Tux
    Jul 16, 2011 at 23:46
  • As stated in the question, these site folders will contain 0 or more files which extend a common file. I need to find out which sites are extending this common file. MY_Output.php resides out of this directory entirely. The files inside these site level folders, if they extend it, will contain the text "extends MY_Output" Jul 16, 2011 at 23:56

8 Answers 8

20
+100

This would be my approach:

find dev -type f -print0 | \          # find all files
xargs -0 grep 'extends MY_Output' | \ # search for your string
cut -d/ -f2  | \                      # extract web folder name
sort | uniq                           # eliminate duplicates

Note use of the print0 parameter to find and the -0 (zero) flag to xargs, which prevents problems if your filenames have embedded spaces in them.

7
  • Dude. You rock so hard. The first 20 minutes or so were spitting out permission denied errors from tmp files but in the end was a solid list of all the site folders I cared about. Booya +1 and accepted. Jul 19, 2011 at 2:34
  • Aww tried to +1 you but I spent all my rep on the bounty haha. Next time ;) Jul 19, 2011 at 2:35
  • 1
    Excellent solution! For completeness, I would have used xargs dirname instead of cut and instead of sort | uniq just sort -u. But there's more than one way to do it :-) Jul 19, 2011 at 7:38
  • @Janne dirname would have caused at least one extra stat call to the file system for each matching file.
    – Alnitak
    Jul 19, 2011 at 8:14
  • @Alienwebguy, you didn't have to suffer through permission denied errors, if you used a script that can deny them... furthermore, you can reduce search speed many times over by caching results. Jul 19, 2011 at 11:11
4

I see a lot of needless complexity in the various solutions posted. Consider the following:

grep -r [regex to find] [path to search] | awk -F: '{print $1}' | uniq
3
  • You can also use cut in place of awk for even greater simplicity - I just happen to remember the existence of awk (and its syntax) more readily than cut...
    – voretaq7
    Jul 22, 2011 at 17:29
  • 2
    You could potentially save a lot of time using the -l argument to grep so it stops reading through files after it finds a match and moves on. That's easy. Some of the more complex answers here are a whole 'nother level more efficient because they stop scanning a directory after the first match is found.
    – Caleb
    Aug 5, 2011 at 18:20
  • 2
    +1 for -l (an option I consistently neglect). I wasn't 100% sure if the OP wanted just the directories or the files but it sounded like they would need to edit individual files in each directory, which is why I went with listing each file. Much time can be saved by bailing if you only need the directory names :)
    – voretaq7
    Aug 5, 2011 at 20:11
3

To find out which websites have a file with MY_output.php inside them follow these steps:

Log into the server using PuTTY. You should get a command prompt like this:

username@hostname:~$

Change to your dev directory (not sure where this is on your server, maybe under /var/www ?)

cd /var/www/dev

Make sure we're in the correct directory

ls

And we should get something like this:

sitea.com
siteb.com
sitec.com

Now find every file which contains MY_output.php then shorten list to just the domains:

grep -rsl "extends MY_output" * | cut -d/ -f1 | sort | uniq

Grep looks for all matching files recursively in all the web directories. The cut command breaks up your paths and just gives the first directory name (sitea.com, siteb.com, etc.) The sort and uniq commands just remove duplicate entries so each domain only appears once.

Enjoy!

7
  • This looks promising. The issue though is these site folders won't have that file - MY_Output.php is a common file. The site folders will have files which extend it, hence why I need to find the string MY_Output inside the contents of the files.
    – AlienWebguy
    Jul 15, 2011 at 19:44
  • 1
    Can I change it to grep -r "MY_output" * | cut -d/ -f1 | sort | uniq ?
    – AlienWebguy
    Jul 15, 2011 at 19:45
  • Just a follow up, that didn't work either. Jul 16, 2011 at 23:32
  • Also tried adding the case-insensitive recursive flag: grep -iR Jul 16, 2011 at 23:33
  • Can you include a few lines of the files from the files which contain the "MY_output" text along with their path? Also, try running just the first part of the command: grep -r "MY_output" * to see if that outputs anything
    – secretmike
    Jul 17, 2011 at 1:49
1

use find to list all files and then run grep on each one of them:

find dev/ | xargs grep -l MY_Output.php 

Note: you must be in the directory containing the dev directory

and if you get errors of unreadable files (permissions etc) then you can append 2>/dev/null

find dev/ | xargs grep -l MY_Output.php 2>/dev/null
3
  • Putty just responds with (standard input). Am I missing something obvious?
    – AlienWebguy
    Jul 14, 2011 at 19:17
  • hey, i updated the answer. But honestly i have no idea why you get such error. maybe you should try if find dev/ gives you all the files in dev directory first. With pipe the grep is using this output as its standard input.
    – Imre L
    Jul 14, 2011 at 22:33
  • Neither worked unfortunately - the first method returned a bunch of random images, psd's, and permission denied messages, and the 2nd method gave me permission denied
    – AlienWebguy
    Jul 15, 2011 at 0:09
0

If you log into your server and then cd to the dev folder, this should work:

egrep -rl 'class \S+ extends MY_Output' * | awk -v FS="/" '{print $2}' | sort | uniq -c

This will pick up ONLY sites that actually do have classes that extend the MY_Output class, and it will also give you a count of files within that site that you can expect to change. You can also use the cut utility instead of awk:

egrep -rl 'class \S+ extends MY_Output' * | cut -d/ -f2 | sort | uniq -c
1
  • AlienWebguy said dev has several hundreds of directories. Using file expansion is not the best approach because the expanded list may be longer than the maximum command line length allowed.
    – joechip
    Jul 18, 2011 at 5:04
0

As I understand it, in general terms you want to list which folders contain regular files called dev/*/* which themselves contain the string "MY_Output" (case sensitive).

More precisely, since the "*" pattern does not match files or directories starting with a ".", you would also want to search for dev/*/*, dev/.??*/*, dev/*/.??* and dev/.??*/.??* . This is a detail that is missing from secretmike's and Brian Showalter's solutions.

Using these four patterns ensures that you process all the files you want and that you don't get extra matches from other (deeper or shallower) files. Normally the list of matching files can be accomplished simply by running:

grep -l MY_Output dev/*/* dev/.??*/* dev/*/.??* dev/.??*/.??* 2>/dev/null 

The 2>/dev/null part is there to ignore errors such as when you try reading from files without permissions, like you seem to be doing (based on your answer to Imre L's answer). For best results you may want to run this command as root.

Unfortunately there's a limit to command line lengths, and this command may fail if there are too many matching files because the command line would be too long (after expansion). Since you say there's hundreds of directories under dev, this approach is not appropriate, although I figure it's worth mentioning here for completeness.

To avoid that problem, the find command is better suited:

find dev -mindepth 2 -maxdepth 2 -type f \
    -exec bash -c 'grep -q MY_Output {} && echo {}' \;

This is very similar to the answer Imre L gave, although he resorts to xargs instead of using the -exec switch. But that only gets you the list of matching files, not the list of folders that contain those.

To get what you want we need to filter it a bit further:

find dev -mindepth 2 -maxdepth 2 -type f \
    -exec bash -c 'grep -q MY_Output {} && dirname {}' \; \
        | sort \
        | uniq

The -mindepth and -maxdepth switches ensure we don't get matches from files deeper or shallower in the dev tree.

0

I think I'd do something like this:

for dir in dev/* ; do
    if [ -d "$dir" ] ; then
        if [ -n "$( find "$dir" -type f \
            -exec grep -q '\<MY_Output\>' {} \; \
            -print | head -1 )" ] ; then
            echo "$dir"
        fi
    fi
done

Or, as a one-liner:

for dir in dev/* ; do [ -d "$dir" ] && [ -n "$( find "$dir" -type f -exec grep -q '\<MY_Output\>' {} \; -print | head -1 )" ] && echo "$dir" ; done
0
find (Directory) -iname "(File Pattern>)" -exec zgrep "(String to Find)" {} \;

For example:

find /opt/WebSphere/AppServer/profiles/application/logs/ -iname "SystemOut*" -mtime -7 -exec zgrep "FileNotFoundException" {} \;

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .