Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have sets of verbose logfiles that in the course of solving a problem, I will repeatedly regrep.

I usually have about 1-10GB sets of 50-150 files, that I'll spend a few hours with and then never look at again.

Even with an SSD and lots of RAM it can take a few dozens of seconds to get results. It also only pegs 1 core, so if it could search in parallel, that would be good too.

I'm wondering if I can do any better by indexing in some way. It would be nice to spend a few minutes up front to have better performance later.

Preferably it would be something I can run at the terminal in the directory, and have an interface like grep. Then at the end I can delete the folder entirely and that will also delete the index.

Does this sound possible, and does something exist? What's my next best option?

share|improve this question
    
Best I can suggest is anytime to run grep on a file [or set of], send the output to a new file and do further narrowing on that smaller 'pre-grepped' file. –  lVlint67 Sep 20 '13 at 16:01
    
Are you searching for different results each time or are you re-using older queries and grabbing new data with it? –  Andrew Domaszek Sep 20 '13 at 16:08
    
unix.stackexchange.com/a/66297/22470 –  mtm Sep 20 '13 at 17:09

1 Answer 1

up vote 0 down vote accepted

Your best bet is probably more complicated than you are willing to set up, given your requirements, such that they are.

Use a logging aggregation stack that can read/tail the files for you ( fluend, index them Elastic Search and present a pretty interface Kibana for you.

Just configure it to delete them as often as you like.

this is only one solution stack, check out logstash as well as many others

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.