Tool to search text in big files

Question

I'm looking for a tool to find a line containing a searced text inside a file that has a size of 4 GB

Answer 1

When you use *nix, you could also use split to get smaller files and then parse it with whatever you like, be it grep, awk, ...

Answer 2

If you have powershell installed you could use

select-string -pattern <your_string> -path <path_to_file>

It probably won't be fast, but won't choke like find or findstr probably will.

Answer 3

Use grep. Nothing will be any fast.

Answer 4

Not free, but if this is a text file, then BareTailPro might do the job. I have used it to search for text in logfiles which were too large to fit in memory. One advantage is that it does not just show the text, but allows you to jump into the text file, so you can see the lines before and after the hits.

Answer 5

If you do it often enough, and the file is broken into logical lines, you could load it into Splunk and search from there. It'll do indexing for you to be quick(er).

Answer 6

I'd use grep on *nix, and I'd use a higher end text editor (e.g Notepad++) on Windows.

Answer 7

It depends on the application, the response time needed, and what you're willing to do to meet those goals.

Recently, I was working with a 10+ GB, 50+ million line text file and had a need to search for specific strings in each line. Standard Unix tool "grep" did the trick, but took an unacceptably long time (multiple minutes). I imported the text into a postgreslq DB (it was a CSV file, easily imported), and once indexed on the key I needed to search on, it took under 1 second to find my record.

Granted, my workstation is single-core, with only 4GB RAM, a 4-year-old 2GHz CPU, and a top-heavy filesystem (ZFS) using 5+ year-old consumer PATA drives. Your mileage will certainly vary. Still, the time difference between the two methods is staggering.

If your data is free-form text, you might still consider importing into a DB which supports full-text search and indexes appropriately to support such searches.

Even if you have the RAM to have the entire file cached and a fast machine, doing a linear search of files this size will be time inefficient, depending (once again) on the application.

Answer 8

At the Windows CMD prompt, there are two commands FIND and FINDSTR. They will probably choke on a file that size or be very slow, but you already have them.

Type help findstr and help find for documentation.