29

Folks, is there a *nix solution which would make the log file act as a circular buffer? For example, I'd like log files to store maximum 1Gb of data and discard the older entries once the limit is reached.

Is it possible at all? I believe in order to achieve that a log file should be turned into some sort of special device...

P.S. I'm aware of misc logrotating tools but this is not what I need. Logrotating requires lots of IO, happens usually once a day while I need a "runtime" solution.

3
  • 3
    I am not sure why you think log rotation would need lots of IO. Rotating 10 logfiles is 10 rename operations and a HUP of the service. Not exactly a murderous operation... And it's the standard solution to your problem :)
    – pehrs
    Apr 17, 2010 at 8:36
  • 2
    One may be running a script/executable which doesn't play well with HUP.
    – Scott
    Oct 17, 2011 at 17:08
  • 1
    This is to provide another use case for your question. I have a demonised music player. I want a log of length just several lines, so I can see what is playing, and what played before that. A tail -f somefile would do that. I just tried with rotated logs and tail -f doesn't work with those.
    – Vorac
    Jun 19, 2018 at 18:51

6 Answers 6

19

Linux has a kernel ring buffer. You can use dmesg to display it.

Or here is a Linux kernel module that appears to do what you want.

What is emlog?

emlog is a Linux kernel module that makes it easy to access the most recent (and only the most recent) output from a process. It works just like "tail -f" on a log file, except that the storage required never grows. This can be useful in embedded systems where there isn't enough memory or disk space for keeping complete log files, but the most recent debugging messages are sometimes needed (e.g., after an error is observed).

The emlog kernel module implements simple character device driver. The driver acts like a named pipe that has a finite, circular buffer. The size of the buffer is easily configurable. As more data is written into the buffer, the oldest data is discarded. A process that reads from an emlog device will first read the existing buffer, then see new text as it's written, similar to monitoring a log file using "tail -f". (Non-blocking reads are also supported, if a process needs to get the current contents of the log without blocking to wait for new data.)

2
  • 1
    Thanks for the link! BTW, emlog home page has a link to ulogbufd which is probably even a more appropriate solution for me.
    – pachanga
    Apr 17, 2010 at 13:30
  • The emlog kernel module is now maintained on github: github.com/nicupavel/emlog
    – dbernard
    Jul 7, 2017 at 18:44
5

The closest thing I can think of is RRDTools, but probably it is not what you are looking for. Another solution would be to monitor the log file (say every second or in Linux with inotify), e.g. you write a script like:

while :; do
  if [[ $(stat -c %s $FILE) -gt 10000 ]]; then
    # rotate the log
  fi
  sleep 1
done

with inotify:

while :; do
  if inotifywait [some options] $FILE; then
    # check size and rotate the file
  fi
done
2
  • +1 for mentioning RRDtool, a real example of ring data structure logging.
    – Cory J
    Apr 17, 2010 at 18:53
  • Thanks for example showing inotifywait shell command usage
    – pachanga
    Apr 20, 2010 at 4:51
5

You can use multilog from djb's Daemontools. You pipe your log output into it. Yes it's log rotation, but rotations are simply:

ln current $tai64nlocaltimestamp

Which, on just about any modern linux filesystem is a super fast operation. You can specify how many log files you want, how big you want them. make 10 x 1024mb files, and you'll have your 1gb ring buffer.

Note, that because of the automatic rotation, it's one source per multilog instance. But you can work around that by writing a simple wrapper with netcat or by hand.

1
  • Thanks for the tip! I'm definitely going to have at multilog as well.
    – pachanga
    Apr 17, 2010 at 13:32
1

You could make a FIFO pipe and then from that read it using a script that inserts to a database. When the counter reaches 1,000, restart the id number being inserted to the database. Wouldn't work for size of course, but you used that as an example, so I'm assuming this is a theoretical question.

0
1

Interesting question; you don't usually see that as a design. I do have a program that uses a faintly similar technique to record history, but it uses a binary format. The 'log file' has four parts, all laid out in a machine-neutral format:

  1. A header containing the magic number and the (maximum) number of entries in the used list and free list, the sequence number for the next history entry, the actual number of entries in the used list, the actual number of entries in the free list, and the length of the file (each of which is 4 bytes).
  2. The used list, each entry giving an offset and a length (4 bytes for each part of each entry).
  3. The free list, each entry similar to the used list entry.
  4. The main data, each history record consisting of a contiguous set of bytes terminated by a null terminator byte.

When a new record is allocated, if there is space in the free list, then it overwrites an entry there (not necessarily using it all - in which case the fragment remains on the free list). When there is no space in the free list, then new space is allocated at the end. When an old record rotates out, its space is moved to the free list, and coalesced with any adjacent free records. It is designed to handle SQL statements so the records can be spread over many lines. This code works on a specified number of records. It does not limit the size of the file per se (though it would not be hard to make it do so).

The main code history code is in two files, history.c and history.h, available from the source for the program SQLCMD (my version, not Microsoft's; mine was in existence a decade or more before Microsoft's), which can be downloaded from the International Informix User Group's Software Archive. There is also a history file dump program (histdump.c) and a history tester (histtest.ec - it claims to be ESQL/C, but is itself really C code; one of the support functions it calls uses some Informix ESQL/C library functions). Contact me if you want to experiment without using Informix ESQL/C - see my profile. There are some trivial changes to make for it to compile histtest outside its design milieu, plus you need a makefile.

0

I agree with pehrs' comment to your question. Log rotation is not that hard. You can set up logrotate or another script to periodically check your log file, even as often as every minute if you so wish. When it detects your file reaches 1GB in size, it simply performs a rename, which takes next to no I/O. During the rename the process continues writing the log file. The log rotator can then send a HUP to your syslog daemon (your daemon is logging through syslog, right? If not, it should support the HUP signal if it's well-written...) to have it re-open the original file path. At this point it will start writing to a new file at the original path and you can delete the rotated version.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .