I need to search something in a huge log-file (over 14 GB). I'm pretty sure it's in the last 4 GB or so.
Is there a way to skip the first X GB to speed things up?
Server Fault is a question and answer site for system and network administrators. It only takes a minute to sign up.Sign up to join this community
I guess you could use tail to only output that last 4GB or so by using the
output the last NUM bytes; or use -c +NUM to output starting with byte NUM of each file
You could probably do something with dd too by setting
skiping to the offset you want to start e.g.
dd if=file bs=1024k skip=12g | grep something
I'm just posting this because some of the comments asked for it.
What I end-up using was (15 GB file). It worked very fast and saved me a ton of time.
tail -f -c 14G file | grep something
I also did a very rudimentary benchmark on the same file. I tested:
grep xxx file
// took for-ever (> 5 minutes)
dd if=file bs=1 skip=14G | grep xxx
// very fast < 1 sec
tail -c 14g | grep xxx
// pretty fast < 2 sec
tail is just a bit shorter.
NB: the suffix used
G differ per command (Ubuntu 15.10)
This doesn't answer the Title question, but it will do what you are wanting to do. Use tac to reverse the file, then use grep to find your string. If your string only occurs once or a known number of times in the file, then let it run until it finds the known number of occurrences. That way, if your assumption about where it is in the file is incorrect, it will still find it. If you do want to limit it, you can use head to do that. The head command would go between the tac and the grep.
So the command looks like:
tac < logfile | grep myString