2

My observation of logrotation is that to rotate any log file, the logrotate process, in the following order

  1. copies the log file in question (lets call it file1.log) with a new name (adding time stamp or a number to the existing name so that it becomes file1.log-20140513),
  2. deletes the existing file (file1.log) and creates a new blank log file with the original name (file1.log),
  3. compresses the rotated file (file1.log-20140513), which creates a new compressed file (file1.log-20140513.gz), if compression option is set,
  4. deletes the rotated file (file1.log-20140513), and the finally,
  5. move to the next file to do the same above 4 steps to it.

I am having the following problem with this process:

  1. My log files are huge in size (more than 10 Gb each) and I have around 42 such log files.
  2. The processes which write to these files work in sync,
  3. The DiskIO on the server is good, but still, copying takes time, and compressing also takes time and compression also consumes CPU.
  4. I want all the newly created log files to have logs starting from same time.

To do that, I want logrotate to move, some thing which the mv command does, which renames the files instead of copying them. As for compression, I can disable that and trigger it via a different script scheduled via cron. But I want logrotate to move the files instead of copying them.

Now, am sure this is some thing the authors of logrotate too would have thought as it obviously saves disk IO and time taken to complete the entire logrotate operation, so I want to why files are being copied instead of being moved or renamed, and how do I achieve this via logrotate.

NOTE: I tried to do it manually, which was, move the file to which a running process was writing too, and create a new blank file with same name and same permissions (which is root, which is also the permission which the process is running with), but after moving and creating a new file, I saw that the process was not writing anything to it, so had to restart the process to make it write to that file. Can any one explain this behaviour too as to why logrotate manages to make the process write to the same file but I'm not able to using the simple steps.

1

The behavior you are describing only happens if logrotate has been explicitly told to do so through the copytruncate directive. The documentation warns about the possibility to lose some log data due to this behavior. That directive should only be used as a last resort.

The standard method of rotating logfiles is to rename and then send a signal to the process to let it open the new logfile. This is faster and doesn't risk losing part of the log. But it requires the writing process to be able to switch to a new logfile.

The compression can be turned off or postponed until the next rotation. If the compress directive is used, the old logfiles are compressed. If that directive is not used, they are not compressed.

If both compress and delaycompress are used, compression is delayed until the next rotation. That way after each rotation the two newest logfiles will not yet have been compressed.

0

after moving and creating a new file, I saw that the process was not writing anything to it, so had to restart the process to make it write to that file

Process is writing to the same file, thats why lograte copy instead of move. When you delete log, process still writes to log and you can see filesystem usage growing, but no file. Process restart will free disk space.

consider to

  • Even though log rotate copies to a new file, it does deletes it after copying and creates a new file to which the process continues to write. How is that achieved? Yes, those logs are important as the application is undergoing a massive up gradation – Gautam Somani May 13 '14 at 16:20
  • 1
    to shrink file try echo "" >/path/to/logfile – Ľubomír Mlích May 13 '14 at 16:22
  • perhaps rotating logs hourly will do what you want – Ľubomír Mlích May 13 '14 at 16:23
  • Shrinking is not what I want. We want all the logs. Hourly is not feasible as analyzing them later would need us to merge them. We simply remove the mismatched lines and insert them into correct files, but we don't want to the delay in rotation to happen in first place. – Gautam Somani May 13 '14 at 16:32
  • Yes, I know you don't want to shrink logs. Just tried to answer your question. Maybe not quite right. I think you ask more than one question and if You divide them, each answer can be found link. Kasperd's answer is more accurate than mine. – Ľubomír Mlích May 13 '14 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.