9

Ok, this is a very practial use case from my point of view.

Let say I have a some simple shell oneliner which does log the output into a file. This can be simply anything, for example tcpdump. Is there any generic and trivial way, to make sure, that the output file won't exceed given size?

Resoning behind this, is to protect from filling the whole available space on the mount point by mistake. If I forget about the script, or it will yield GBs of data per hour, then this simple debugging task can lead to a potential system crash.

Now, I am aware of the options build in some of the tools (like combination of -W/-C in tcpdump). What I need is a very generic failsafe.

Long story short - when I run a script like:

% this -is --my=very|awsome|script >> /var/tmp/output.log

How to make sure that output.log will never get bigger than 1GB.

Script can crash, be killed or whatever.

Solution I am looking for should be easy and simple, using only tools available in popular distros like ubuntu/debian/fedora. In general something widely available. Complicated, multiline program is not an options here regardless of the language/technology.

15

You can use head for this:

command | head -c 1G > /var/tmp/output.log

It accepts K, M, G and the like as suffixes (bytes are the default). Append 'B' to use the base 10 versions.

  • head agains wall It couldn't be more simple. Thanks! – mdrozdziel Jun 7 '11 at 18:31
  • 2
    It seems tail -c 1G also works, useful to look at the tail of an event. Understandably it flushes after the command end. – dashesy Oct 7 '16 at 0:05
  • Unit for suffixes do not work on Mac OS. – anumi Jan 22 at 15:15
0

Set the maximum file size for a user that will only be used to run these scripts.

The file /etc/security/limits limits a user with the values of "default" unless there are explicit values for a specific user. These user specific values will overwrite the default values. The file may have a slightly different name depending on your OS.

If your log user is named log_maker, then add this line to the file:

log_maker hard fsize 1000000

The number after fsize is the maximum file size in KB.

0

curtail limits the size of a program's output and preserves the last X KB of output with the following command:

run_program | curtail -s 1G myprogram.log

https://github.com/Comcast/Infinite-File-Curtailer

NOTE: I'm the maintainer of the above repo. Just sharing the solution...

0

You can limit any file with

tail -c 1G myLog.log > myLog.tmp
mv myLog.tmp > myLog.log

Writing the tail directly to the same file makes an empty file. So you need to use a temporary file.

  • tail -c 1G myLog.log > myLog.tmp mv myLog.tmp > myLog.log this will kill the log, i think you wanted to use >> instead of > or dont you? – djdomi Aug 7 at 10:23
  • The purpose of this command is to rewrite the log with only the last 1G. The >> should be for the previous command : doSomething >> myLog.log; tail -c 1G myLog.log > myLog.tmp; mv myLog.tmp > myLog.log; – Séverin Aug 7 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.