Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I need to run a grep on my server to search for files which extend a CodeIgniter 1 file, as we are upgrading to CI2. In one folder contains several hundred site-specific folders like such:

dev/sitea.com/site/www
dev/siteb.com/site/www
dev/sitec.com/site/www
... and so on

How can I grep to find which folders directly under dev contain a file (or many files in variable subfolders) which references the string "MY_Output" (php file)

I'm aware of the -l flag to list the files, but I'm not sure how to put it with other flags together to say "show me which websites contain files with this string"

Is this possible? Thanks!

EDIT: Just to clarify, these site level folders will contain files which extend MY_Output.php, which is a common shared file:

class Whatever extends MY_Output
share|improve this question

migrated from stackoverflow.com Jul 16 '11 at 3:30

This question came from our site for professional and enthusiast programmers.

    
This is not related to apache, ssh or putty (thus I removed the tags). –  Paŭlo Ebermann Jul 15 '11 at 18:42
    
I'm using Putty to run these greps. Figured it may be applicable. –  AlienWebguy Jul 15 '11 at 19:42
2  
Maybe bash could be applicable (as this is most likely the shell you are executing the command in), but the command should not depend on how you connect. (Please do not regard this as an attack, I just wanted to mention why I removed the tags, in case anyone wonders.) –  Paŭlo Ebermann Jul 15 '11 at 19:45
    
Yep no worries, thanks for the heads up :) –  AlienWebguy Jul 15 '11 at 19:52
    
Perhaps you could include something about what a positive match might look like. For instance would the file /dev/sitea.com/site/www/file contain the text "MY_Output" within it, or are you looking for "MY_Output" within the path itself? –  Red Tux Jul 16 '11 at 23:46

9 Answers 9

up vote 16 down vote accepted
+100

This would be my approach:

find dev -type f -print0 | \          # find all files
xargs -0 grep 'extends MY_Output' | \ # search for your string
cut -d/ -f2  | \                      # extract web folder name
sort | uniq                           # eliminate duplicates

Note use of the print0 parameter to find and the -0 (zero) flag to xargs, which prevents problems if your filenames have embedded spaces in them.

share|improve this answer
    
Dude. You rock so hard. The first 20 minutes or so were spitting out permission denied errors from tmp files but in the end was a solid list of all the site folders I cared about. Booya +1 and accepted. –  AlienWebguy Jul 19 '11 at 2:34
    
Aww tried to +1 you but I spent all my rep on the bounty haha. Next time ;) –  AlienWebguy Jul 19 '11 at 2:35
    
Excellent solution! For completeness, I would have used xargs dirname instead of cut and instead of sort | uniq just sort -u. But there's more than one way to do it :-) –  Janne Pikkarainen Jul 19 '11 at 7:38
    
@Janne dirname would have caused at least one extra stat call to the file system for each matching file. –  Alnitak Jul 19 '11 at 8:14
    
@Alienwebguy, you didn't have to suffer through permission denied errors, if you used a script that can deny them... furthermore, you can reduce search speed many times over by caching results. –  Mike Pennington Jul 19 '11 at 11:11

I see a lot of needless complexity in the various solutions posted. Consider the following:

grep -r [regex to find] [path to search] | awk -F: '{print $1}' | uniq
share|improve this answer
    
You can also use cut in place of awk for even greater simplicity - I just happen to remember the existence of awk (and its syntax) more readily than cut... –  voretaq7 Jul 22 '11 at 17:29
1  
You could potentially save a lot of time using the -l argument to grep so it stops reading through files after it finds a match and moves on. That's easy. Some of the more complex answers here are a whole 'nother level more efficient because they stop scanning a directory after the first match is found. –  Caleb Aug 5 '11 at 18:20
1  
+1 for -l (an option I consistently neglect). I wasn't 100% sure if the OP wanted just the directories or the files but it sounded like they would need to edit individual files in each directory, which is why I went with listing each file. Much time can be saved by bailing if you only need the directory names :) –  voretaq7 Aug 5 '11 at 20:11

To find out which websites have a file with MY_output.php inside them follow these steps:

Log into the server using PuTTY. You should get a command prompt like this:

username@hostname:~$

Change to your dev directory (not sure where this is on your server, maybe under /var/www ?)

cd /var/www/dev

Make sure we're in the correct directory

ls

And we should get something like this:

sitea.com
siteb.com
sitec.com

Now find every file which contains MY_output.php then shorten list to just the domains:

grep -rsl "extends MY_output" * | cut -d/ -f1 | sort | uniq

Grep looks for all matching files recursively in all the web directories. The cut command breaks up your paths and just gives the first directory name (sitea.com, siteb.com, etc.) The sort and uniq commands just remove duplicate entries so each domain only appears once.

Enjoy!

share|improve this answer
    
This looks promising. The issue though is these site folders won't have that file - MY_Output.php is a common file. The site folders will have files which extend it, hence why I need to find the string MY_Output inside the contents of the files. –  AlienWebguy Jul 15 '11 at 19:44
    
Can I change it to grep -r "MY_output" * | cut -d/ -f1 | sort | uniq ? –  AlienWebguy Jul 15 '11 at 19:45
    
Just a follow up, that didn't work either. –  AlienWebguy Jul 16 '11 at 23:32
    
Also tried adding the case-insensitive recursive flag: grep -iR –  AlienWebguy Jul 16 '11 at 23:33
    
Can you include a few lines of the files from the files which contain the "MY_output" text along with their path? Also, try running just the first part of the command: grep -r "MY_output" * to see if that outputs anything –  secretmike Jul 17 '11 at 1:49

use find to list all files and then run grep on each one of them:

find dev/ | xargs grep -l MY_Output.php 

Note: you must be in the directory containing the dev directory

and if you get errors of unreadable files (permissions etc) then you can append 2>/dev/null

find dev/ | xargs grep -l MY_Output.php 2>/dev/null
share|improve this answer
    
Putty just responds with (standard input). Am I missing something obvious? –  AlienWebguy Jul 14 '11 at 19:17
    
hey, i updated the answer. But honestly i have no idea why you get such error. maybe you should try if find dev/ gives you all the files in dev directory first. With pipe the grep is using this output as its standard input. –  Imre L Jul 14 '11 at 22:33
    
Neither worked unfortunately - the first method returned a bunch of random images, psd's, and permission denied messages, and the 2nd method gave me permission denied –  AlienWebguy Jul 15 '11 at 0:09

You might be able to do this with grep, but I think the required logic justifies using a script... This is a quick python script that will search based on the parameters you provided above...

First it recursively searches from ROOT_DIRECTORY for any files matching FILE_FILTER... then it searches each of those files for a string matching SEARCH_STRING. If it finds any file matching the SEARCH_STRING it will log the match, and immediately skip remaining files in that directory, moving on to the next file in all_files that hasn't already had a match (saving you some CPU and disk wear).

If I have made incorrect assumptions, you can edit the variables named FILE_FILTER, SEARCH_STRING, and ROOT_PATH.

Save the script below as searchme.py and execute it with python searchme.py

===

import os
import re
import sys

def get_directory(path):
    return '/'.join(path.split('/')[0:-1])

FILE_FILTER = '(\.htm|\.php)'
SEARCH_STRING = 'MY_Output'
ROOT_PATH = '~/'

all_files = []
retval = {}
rootdir = os.path.expanduser(ROOT_PATH)

## Find all files matching FILE_FILTER
for root, subFolders, files in os.walk(rootdir):
    for file in files:
        pname = os.path.join(root,file)
        if re.search(FILE_FILTER, pname) is not None: all_files.append(pname)
        retval[get_directory(pname)] = False

## Search files for SEARCH_STRING; take shortcut if string is found
for pname in all_files:
    path = get_directory(pname)
    if not retval[path]:
        try:
            for line in open(pname):
                if SEARCH_STRING in line:
                    retval[path] = True; break
        except IOError:
            ## Occasionally firefox makes lock files that can't be opened...
            ##     Ignore any permission errors...
            pass

## Print resultant directories...
for path in sorted(retval.keys()):
    if retval[path]: print path
share|improve this answer
    
I understand that such a script would be useful if this search were to be run frequently and over lots (and I mean lots) of files matching dev/*/*. Otherwise, a one-liner makes more sense IMHO. –  joechip Jul 18 '11 at 5:11
    
A script makes sense when it can speed execution by several times over a standard grep... that is precisely what this does –  Mike Pennington Jul 18 '11 at 19:25

If you log into your server and then cd to the dev folder, this should work:

egrep -rl 'class \S+ extends MY_Output' * | awk -v FS="/" '{print $2}' | sort | uniq -c

This will pick up ONLY sites that actually do have classes that extend the MY_Output class, and it will also give you a count of files within that site that you can expect to change. You can also use the cut utility instead of awk:

egrep -rl 'class \S+ extends MY_Output' * | cut -d/ -f2 | sort | uniq -c
share|improve this answer
    
AlienWebguy said dev has several hundreds of directories. Using file expansion is not the best approach because the expanded list may be longer than the maximum command line length allowed. –  joechip Jul 18 '11 at 5:04

As I understand it, in general terms you want to list which folders contain regular files called dev/*/* which themselves contain the string "MY_Output" (case sensitive).

More precisely, since the "*" pattern does not match files or directories starting with a ".", you would also want to search for dev/*/*, dev/.??*/*, dev/*/.??* and dev/.??*/.??* . This is a detail that is missing from secretmike's and Brian Showalter's solutions.

Using these four patterns ensures that you process all the files you want and that you don't get extra matches from other (deeper or shallower) files. Normally the list of matching files can be accomplished simply by running:

grep -l MY_Output dev/*/* dev/.??*/* dev/*/.??* dev/.??*/.??* 2>/dev/null 

The 2>/dev/null part is there to ignore errors such as when you try reading from files without permissions, like you seem to be doing (based on your answer to Imre L's answer). For best results you may want to run this command as root.

Unfortunately there's a limit to command line lengths, and this command may fail if there are too many matching files because the command line would be too long (after expansion). Since you say there's hundreds of directories under dev, this approach is not appropriate, although I figure it's worth mentioning here for completeness.

To avoid that problem, the find command is better suited:

find dev -mindepth 2 -maxdepth 2 -type f \
    -exec bash -c 'grep -q MY_Output {} && echo {}' \;

This is very similar to the answer Imre L gave, although he resorts to xargs instead of using the -exec switch. But that only gets you the list of matching files, not the list of folders that contain those.

To get what you want we need to filter it a bit further:

find dev -mindepth 2 -maxdepth 2 -type f \
    -exec bash -c 'grep -q MY_Output {} && dirname {}' \; \
        | sort \
        | uniq

The -mindepth and -maxdepth switches ensure we don't get matches from files deeper or shallower in the dev tree.

share|improve this answer

I think I'd do something like this:

for dir in dev/* ; do
    if [ -d "$dir" ] ; then
        if [ -n "$( find "$dir" -type f \
            -exec grep -q '\<MY_Output\>' {} \; \
            -print | head -1 )" ] ; then
            echo "$dir"
        fi
    fi
done

Or, as a one-liner:

for dir in dev/* ; do [ -d "$dir" ] && [ -n "$( find "$dir" -type f -exec grep -q '\<MY_Output\>' {} \; -print | head -1 )" ] && echo "$dir" ; done
share|improve this answer
find (Directory) -iname "(File Pattern>)" -exec zgrep "(String to Find)" {} \;

For example:

find /opt/WebSphere/AppServer/profiles/application/logs/ -iname "SystemOut*" -mtime -7 -exec zgrep "FileNotFoundException" {} \;
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.