7

I'm trying to find a simple way to append/write to a log, while at the same time keeping it the log trimmed to a reasonable size. I would prefer to not just append files forever, and then have to have a clean up log script. I can't wrap my head around how I would accomplish this gracefully without using some second file as a temporary holder.

Things I've looked at for reference:

I've gone through the advanced scripting guide http://tldp.org/LDP/abs/html/io-redirection.html

Combined output from two commands: Bash - How to output two commands to file?

Ideally I would have something like (I know this doesn't work)

(tail-n 1000 foo.log; ./foo.sh) > foo.log

which would keep the last 1000 lines from my on going log and then append my new output for the current run of foo.sh.

I can't think of a way to use the append redirect >> and limit the original file without wrapping the call to foo.sh in some other bar.sh

tail -n 1000 foo.log > tmp.log
mv tmp.log foo.log
./foo.sh >> foo.log

This just seems kludgey.

Perhaps my answer is to have foo.sh not rely on STDOUT as a place to send log messages, but rather opens the file directly.

FOLLOW UP EDIT:

The prevailing opinion is that this is not recommended. An opinion that I appreciate. However, the server this is going on is outside of my control and not really going to be under a.... vigilant administrator. Its a box where lots of different groups own parts of the box, but no one is responsible for the total health of the box. I could just let the log build forever, and it probably wouldn't matter in the fullness of time, but I'd like to do what I can to reign it in, since i know the final admin won't do anything. So, using crontab to run logrotate is out for me. I'm just looking for something I can do with the limited point of a single command.

4
  • You would use tail rather than head if you want to keep the latest messages. Sep 2, 2010 at 19:54
  • yes.. tail. Edited post.
    – Marc
    Sep 2, 2010 at 20:10
  • Huh? Any solution will require doing something, right? You can't just think of a solution and will the changes to occur on the server. Either you or the sysadmin would have to touch the server at some point. When that touching occurs touch the logrotate config. Sep 2, 2010 at 22:27
  • I won't be able to convince the sysadmin to touch the logrotate config. I'd be surprised if there was any configuration for logrotate at all. right now I only have one place where I can have another process call a single command where I know call something like 'foo.sh > foo.log' Thats all I have control over. As is, I'm overwriting the log , I can only see the last run. No one cares about this log but me. No monitor is tailing it, I just log in every once in a while and want to see the output from last 100 or so run which would require me to append, but I don't want to just append forever.
    – Marc
    Sep 2, 2010 at 22:59

6 Answers 6

1

If your version of sed has the --in-place switch you could use that. It still does a cp and mv but it's behind the scenes so the script would look a little less messy depending on how you feel about sed syntax. Otherwise your tail (you missed one in your edit) and append method is the way I'd approach it.

Another approach using sed would be to keep your log upside down with the newest entry at the beginning.

sed -ni '1s/^/New Entry\n/;100q;p' logfile

That's all there is to it (other than getting "New Entry" to be what you want). That will keep a rolling 100 line log file in reverse chronological order. The advantage is that you don't have to know how many lines are in the file (tail takes care of this for you, but sed won't). This assumes to some extent that your log entries consist of a single line. If you're logging a significant amount of output from a script I think you're back to tail and append.

Yet another option which might work well for detailed multi-line output would be to let the filesystem do the work for you. Each log entry would be a separate file.

#!/bin/bash
# *** UNTESTED ***
logdir="/path/to/your/log/dir"

# run the script and log its output to a file named with the date and time
script > "$logdir/$(date "+%Y%m%dT%H%M%S").log"

# maintain the logs
# keep the last 100 days of daily logfiles
# (you could keep the last 100 hours of hourly logfiles, for example,
# by using -mmin and doing some math)

# the count is intended to keep old entries from being deleted
# leaving you with nothing if there are no new entries
count=$(find "$logdir" -maxdepth 1 -type f | wc -l)
if (( count > 100 )) 
then
    find "$logdir" -maxdepth 1 -type f -mtime +100 -delete
fi
1
  • Thank you very much for the variety. I had a feeling that sed held a solution, but I'm not particularly versatile with sed. The multi file approach definitely expanded my thinking.
    – Marc
    Sep 3, 2010 at 14:10
1

In general I would recommend you don't worry about the log size. The standard way to manage this is with logrotate. Let the administrators decide how large the want the log to be. Part of being a good *nix app I believe is sticking to the common standards, people get angry when an app does something out of the ordinary.

1

It is inadvisable because there is no way to delete the 1st line of a file without re-writing the entire file. Depending on the file size that is an intensive operation. Even if you did want to do that the two methods for doing so (in place with seeking or unlink, recreate) will result in tail not seeing new lines or thinking all the lines are new.

0

I'd have to agree with Kyle on logrotate et al. But if you are dead set on controlling it from your app, try Googling for something like: linux circular log file

0

Here is a similar post that yielded two good answers for me: Turning a log file into a sort of circular buffer

I had the same problem and I'm happily using ulogbufd.

0

Here's a quick script that I ended up using for this purpose.

It avoids the whole problem of reading and writing at the same time by creating two files, myfile and myfile.old.

#!/bin/bash

# Bail on error
set -e

# Constants
DEFAULT_NUM_LINES="1000"

# Usage message
usage()
{
    echo "Usage: last-lines [-n num] filename" 1>&2
    echo "Options:" 1>&2
    echo "    -n    Specify number of lines file iteration (default ${DEFAULT_NUM_LINES})" 1>&2
    echo "    -h    Show this help message" 1>&2
}

# Parse flags passed in on the command line
NUM_LINES="${DEFAULT_NUM_LINES}"
while [ ${#} -gt 0 ]; do
    case "$1" in
        -n)
            shift
            NUM_LINES="${1}"
            shift
            ;;
        -h|--help)
            usage
            exit
            ;;
        --)
            shift
            break
            ;;
        *)
            break
            ;;
    esac
done
case "${#}" in
    1)
        FILENAME="$1"
        ;;
    *)
        usage
        exit 1
        ;;
esac

# Run
LINE_COUNT="0"
while read LINE; do
    echo "${LINE}" >> "${FILENAME}"
    LINE_COUNT=`expr "${LINE_COUNT}" + 1`
    if [ "${LINE_COUNT}" -ge "${NUM_LINES}" ]; then
        mv "${FILENAME}"{,.old}
        LINE_COUNT="0"
    fi
done

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .