I don't usually do this kind of stuff, so I just have to ask. How do I make a script which checks a log file, lets say it's named log.txt, grabs everything but the first 100 000 lines from it to a new log.txt file and deletes the old log.txt? It would also have to run monthly.

So far with my zero coding knowledge I've only managed to make a script which would delete the log.txt once a month (new one generates automatically), but I would really like to keep data except the oldest 100 000 entries as mentioned below.

Here's the old script

find /mnt/usb_storage/ -type f -name "log.txt" -exec rm -r {} \;

how I implemented it:

# cd /privRoot
# chmod 755 deleteLogMonthly.sh
# crontab -e 

59 23 1 * * deleteLogMonthly.sh
  • 3
    use logrotate for that, it does this. – Dennis Nolte Jul 10 '15 at 8:24

Your are attempting to reinvent wheels.

Logrotate does this efficiently:




Rather than a hard 100K lines, you give it a time interval and it will start writing to a new log file from that period, keeping the old one.

If, for some strange reason, logrotate isn't a true option, check out Cronolog:

Is there any alternate to logrotate for apache logs?


Or indeed manually craft a bash script like this one:


That's still based on days, but you could add some line counting in there.

| improve this answer | |

Add following in your logrotate configuration (new file under /etc/logrotate.d/)

/mnt/usb_storage/log.txt {
           rotate 5 # Change this to number of files you need to preserve
               /usr/bin/sed -e '1,100000d' < /mnt/usb_storage/log.txt
               <some another action>

you can verify it using logrotate -d /etc/logrotate.d/mylogrotate.conf (assuming you have created that file)

| improve this answer | |
  • Okay, I could give this a try, where exactly in logrotate.conf do I add this? – Guy395 Jul 10 '15 at 9:03
  • Better create a new file, if you are just testing. You can create it anywhere and use it for testing purpose. just remember to give correct path when you are testing. Once you are sure that it works as per your requirement, put it under /etc/logrotate.d – Nehal Dattani Jul 10 '15 at 9:07
  • Alright, now I have mylogrotate.conf under loglogrotate.d all configured. Since my system only generates one log file, shouldn't rotate be 2 instead of 5? This is the only part that I don't get :) – Guy395 Jul 10 '15 at 10:07
  • 1
    From man page, rotate count Log files are rotated <count> times before being removed or mailed to the address specified in a mail directive. If count is 0, old versions are removed rather then rotated. – Nehal Dattani Jul 10 '15 at 10:10
  • Alrights thanks, all of you guys have been extremely helpful, now one last question, for testing purposes, how do I make this rotate hourly? Apparently it isn't possible simply by editing myconf, since it gives me an error. – Guy395 Jul 10 '15 at 10:38

lookup the man page of "logrotate" or make it yourself by shell script. there is no magic to it. if you know how to make it by hand, just put it all together in a txt file and you have it.

also check out the man page of the tool "tail". it is designed to display the "tail" or more precisely the latest X lines of a file. "man tail" gives you more information about how to use it.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.