How to truncate the log file size without losing recent logs?

Question

I want to reduce the log file size to the particular minimum size

I tried the truncate command as below

$ truncate -s 20M log filename

But when I check the truncated file it truncates the recent logs and keeps the old logs

When analyzing further I found the tail and head commands. But the problem is when I use the head command is working and when I use tail command it saves the content in the human not readable format like @@@@@@@@@@@.

$ head -c 10M logfile > newfile (it saves the head of the file as mentioned size)

$ tail -c 10M logfile > newfile(It saves the content which cannot be readable)

Now I want to truncate the large file which removes only starting of the file but not the ending of the file and if possible it should be in one command without creating another file as head and tail Can anyone guide me on this? Thank you!

Answer 1

Being a fan of simplicity and high maintainability, I made this. I'm open to your opinion.

This command will truncate log file /tmp/myfile, it cuts the beginning and not the end of the file, it does not require you to configure services and install multi-line scripts; just a one-line command you can add to your crontab.

tail -c 1M /tmp/myfile > /tmp/dzqxH4ZMiSQb91uMMMgPhsgmpnc && rm /tmp/myfile && mv /tmp/dzqxH4ZMiSQb91uMMMgPhsgmpnc /tmp/myfile

First command cuts 1MB of the target file into a temp file, second command deletes the target file, and the third one renames the 1MB temp file as your target file.

It has the disadvantage that it will allocate in your file system, temporarily, 1 MB of space. This could be a problem if you want to keep, say, the last 1 GB of your log file, and your storage is too small or too slow.

If that's your case, you could try this:

fallocate -c -o 0 -l 1M file

It will cut off bytes from your file; starting at position 0, it deallocates 1 MB of disk space off your file. I understand it does it in place, instead of copying data into a temporary file. But it takes more complexity; you would need to first calculate the current size of the file to know how much to deallocate, or make a little loop, something like:

while (file larger than 100M) fallocate 1M; 

resulting in a file between 100 MB and 99 MB. But I like simplicity so for now I'm going with the first option.

Answer 2

The approach: tail -c 10M logfile > newfile should work. Try 'which tail' and 'where tail' to see what is wrong.

I have been using tail -c for many years. Since this is getting a bigger thing for me now, I just now wrote a little tcsh script to do it:

#!/bin/tcsh -f
set echo_style='none' ; setenv LC_ALL C #; set echo verbose

# Purpose: reduce the size of log file FILE to SIZE, if it exceeds
#          twice that amount, by junking the beginning of it.

# Usage: reducelog SIZE FILE
# Clobbers FILE.tmp

# check arguments
set msg="Usage: reducelog SIZE FILE"
if ($#argv != 2) goto err
set size="$1"
set msg="Invalid reduced SIZE"
if (`echo "$size" | grep -c '^[1-9][0-9]*$'` != 1) goto err
set file="$2"
set msg="File not accessible: $file"
if !(-frw "$file") goto err

# check whether we should cut
set msg="Error finding the size of $file"
set oldsize="`ls -l "$file"  | cut -d' ' -f 5`"
if ($status) goto err
if (`echo "$oldsize" | grep -c '^[0-9][0-9]*$'` != 1) goto err
@ maxsize = $size + $size
if ("$oldsize" < "$maxsize") goto end

# cut it down quickly (so that hopefully nothing else interferes)
set msg="Error reducing the size of $file"
tail -c "$size" "$file" >! "$file.tmp"
if ($status) goto err
set msg="Error replacing $file"
mv -f "$file.tmp" "$file"
if ($status) goto err

# check that we did reduce the size
set msg="Error finding the new size of $file"
set newsize="`ls -l "$file"  | cut -s -d' ' -f 5`"
if ($status) goto err
if (`echo "$newsize" | grep -c '^[0-9][0-9]*$'` != 1) goto err
set msg="Failed to reduce the file size"
if !("$newsize" < "$oldsize") goto err

# done
goto end

# error
err:
echo "*** $msg\!"
exit 1

# end
end:

The reason I am still googling this is that I am somewhat concerned about the (remote) possibility that some process writes to the log file just as I am moving it into position. But I guess there is simply no other way to find on the web.

Note: in the script above, SIZE is in bytes, e.g. 100000 for 100 kB.

Answer 3

Configure logrotate to manage your log files.

What you most probably want is the copytruncate option.