I know that head and tail can take -c option to specify a byte offset. I'm looking for a way to efficiently extract a byte range from a large log file.


The DareDevil of the Unix commands, dd to the rescue!

dd if=yourfile ibs=1 skip=200 count=100

That would start from byte 200 and show 100 next bytes, or in other words, bytes 200-300. ibs means dd only reads one byte at a time instead of the default 512 bytes, but still writes out in default 512 byte chunks. Go and see if ibs harms the performance, I hope not.

| improve this answer | |
  • For a file of 782090815 bytes, I got these times: time dd if=file.txt | wc -l = 00:00:03s. time dd if=file.txt ibs=1 count=782090815 | wc -l = 9:05:19 – Danilo Souza Morães Aug 30 '19 at 4:54

If your interest is in the bytes, would od be of more interest.

-j, --skip-bytes=bytes
-N, --read-bytes=bytes

So to read the 16 bytes starting at byte 1024, and output in ascii

od -j 1024 -N 16 -a /bin/sh
| improve this answer | |

You can use dd if=logfile of=pieceoflogfile skip=startingblock count=#ofblocks (possibly with bs=1 to get one-byte blocks, otherwise it uses 512 byte blocks). Not sure how efficient it is to tell it to write one byte at a time, though.

| improve this answer | |
  • Well, if concerned regarding efficiency — 2 dd's can be pipe-chained together (the 1st might cut fat piece and the 2nd would do fine job over the pipe, not disk), but dd has not only bs but separate ibs and obs as well, so at least it could output with bigger blocks than reading. – poige Jul 11 '12 at 15:25

Assuming the file isn't excessively large (e.g. several GB or so), piping from one to the next is as efficient as you'll get, short of writing your own program to do so.

head ... file | tail ...

(Or the other way around. Whichever.)

| improve this answer | |
  • 1
    tail should go first for large offsets, otherwise the beginning of the head output is discarded. – proski Aug 25 '19 at 19:59

I know this is old, and asked for a "linux" command in the title, but python is perfect for this and comes with most linux distributions. It's as easy as this:

python -c 'f=open("myfile.txt","rb");f.seek(100);print(f.read(100));f.close()' | SomeOtherProgram

f.seek(n,0) goes n bytes from the start of the program, and f.read(n) reads the number of bytes you want.

I've not found any other options that make it that easy.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.