Delete a log while is written using SED - /var/log/messages

Question

I had a filesystem that was over 60%. I found that the file /var/log/messages had more than 4 million lines so I decided to delete 2 million lines in order to clear some space. I executed the command

sed -i '1,2000000d' /var/log/messages

the lines were deleted and the log size decreased about 300MB, but the filesystem usage increased from 60% to 90%.

Then I restarted the service rsyslog but nothing happened, the usage is still 90%.

Note that if I execute the command du -sh * and then I sum the usage of each file the total usage does not correspond with the 'Used' value of the filesystem. Seems like there should be a hidden file occupying the rest of the space but I do not know how to find it or delete it.

Answer 1

I guess it's not advisable to delete a file while in use because can cause this type of behavior. I've seen cases in which some users have deleted/reduced big files while the app was still logging to it and the filesystem doesn't think to notice the changes in size until they restarted the app.

In many distros you can use logrotate to rotate this file and avoid the logs to fill up the partition. In RHEL6 you can configure this by editing the /etc/logrotate.conf file and adding a block like this:

/var/log/messages { rotate 4 size 50M postrotate /usr/bin/killall -HUP syslogd endscript }

Explanation:

1. rotate 4: Keep the last four files.
2. size 50M: They are rotated whenever it grows over 50M in size.
3. /usr/bin/killall: command to be executed after rotation.

Please see more info in the man pages: man logrotate.

Note: It's likely you reclaim the space by rebooting the node.

I hope this helps.

Answer 2

Finally I found a solution for this issue.

The problem was that the deleted file was still open by the system

[root]# lsof | grep messages
COMMAND     PID      USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
abrt-dump  2015      root    4r      REG              252,6 417938659        195 /var/log/messages (deleted)
[root]#

so the problem was solved when I closed that file executing the gdb command.

[root]# gdb -p 2015
(gdb ) p close (4)

After executing the commands the file system usage is 25 %

Thanks