0

I had a filesystem that was over 60%. I found that the file /var/log/messages had more than 4 million lines so I decided to delete 2 million lines in order to clear some space. I executed the command

sed -i '1,2000000d' /var/log/messages

the lines were deleted and the log size decreased about 300MB, but the filesystem usage increased from 60% to 90%.

Then I restarted the service rsyslog but nothing happened, the usage is still 90%.

Note that if I execute the command du -sh * and then I sum the usage of each file the total usage does not correspond with the 'Used' value of the filesystem. Seems like there should be a hidden file occupying the rest of the space but I do not know how to find it or delete it.

0

I guess it's not advisable to delete a file while in use because can cause this type of behavior. I've seen cases in which some users have deleted/reduced big files while the app was still logging to it and the filesystem doesn't think to notice the changes in size until they restarted the app.

In many distros you can use logrotate to rotate this file and avoid the logs to fill up the partition. In RHEL6 you can configure this by editing the /etc/logrotate.conf file and adding a block like this:

/var/log/messages { rotate 4 size 50M postrotate /usr/bin/killall -HUP syslogd endscript }

Explanation:

  1. rotate 4: Keep the last four files.
  2. size 50M: They are rotated whenever it grows over 50M in size.
  3. /usr/bin/killall: command to be executed after rotation.

Please see more info in the man pages: man logrotate.

Note: It's likely you reclaim the space by rebooting the node.

I hope this helps.

| improve this answer | |
  • I still have the problem, any other idea?? I tried also with killall -9 but the partition usage does not decrease. – Galo Ramirez Jan 7 '19 at 17:15
  • When you said you still have the problem you meant you rebooted the node and still the Use% column shows 90%? Are you sure there's not any other data that is actually using that space? – Manuel Florian Jan 7 '19 at 17:50
  • Yes, the Use% column shows 90%. I configured the logrotate as recommended but the usage did not decrease. I forced the rotation of the logs using the command logrotate --force /etc/logrotate.conf. Then I executed du -sh * and added the use of each file, the total space used by the files inside the partition is 217 MB, but according to the output of df -h, the space used is 632MB. – Galo Ramirez Jan 7 '19 at 18:34
0

Finally I found a solution for this issue.

The problem was that the deleted file was still open by the system

[root]# lsof | grep messages
COMMAND     PID      USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
abrt-dump  2015      root    4r      REG              252,6 417938659        195 /var/log/messages (deleted)
[root]#

so the problem was solved when I closed that file executing the gdb command.

[root]# gdb -p 2015
(gdb ) p close (4)

After executing the commands the file system usage is 25 %

Thanks

| improve this answer | |
  • thanks for the feedback. But it's weird that the reboot didn't solve the issue. – Manuel Florian Jan 7 '19 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.