Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

What is the most efficient way to perform full-text searches on files in a filesystem?

share|improve this question
1  
... "Binarise"? –  Ignacio Vazquez-Abrams Oct 23 '10 at 2:56
    
"Compile" may be a better word –  user58006 Oct 23 '10 at 2:59
2  
Not really. Index might be. Brute-force search, use your OS's find, grep, etc. If it's a structured file, consider loading it into a database engine like SQLite and then working on it. –  TessellatingHeckler Oct 23 '10 at 3:00
    
This sounds more like a question for Super User of Stack Overflow. –  Cristian Ciupitu Oct 23 '10 at 3:02
1  
This question is overly broad. You should narrow its scope and be more specific. Otherwise find ... -print0 | xargs -0 grep ... or find ... -exec grep ... + are the most general-purpose answers that are efficient enough. Specifying the starting directory and the proper options will do a lot for efficiency. Indexed searches are very fast, but the index has to exist first. It all depends on what you're trying to do. –  Dennis Williamson Oct 23 '10 at 5:16
show 1 more comment

1 Answer

If the number of queries greatly exceeds the number of changes to the files then the most efficient way to perform full-text search is to use a pre-built index that gets updated when files are added, changed or removed.

The type of index needed depends on the type of searches. For example, do you search on words or phrases? Do you want to take into account synonyms? Do you search on word-fragments?

What kinds of efficiencies are you looking for? Development time efficiency? Efficiencies in software licencing costs? CPU time efficiency? Disk IO efficiency? The way you weight these will have a large effect on which solution is optimal for you.

See also Wikipedia

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.