I have a huge log file I need to delete on a production web server. I'm worried it'll bring the system to a crawl if I rm it on Linux. Any brilliant ideas?


Filesystem: ext3

Partition: /var (mostly logs and MySQL data)

Log file is no longer being written to. (No additional data is being appended)

Web Server is LAMP (lots of IO)

  • 1
    Duplicate of serverfault.com/questions/128012/… ? What is your underlying file system? – Ben DeMott Dec 2 '11 at 5:55
  • ext3 - hoping for something more inspiring that ionice might work... I read the man page (linux.die.net/man/1/ionice) and it sounds like the right tool... I'd like a someone with experience using it to chime in. I don't feel very adventurous when it comes to production servers. – objectoriented Dec 2 '11 at 6:06
  • Did any of these solve the problem? – ewwhite Mar 5 '12 at 2:14
  • No, I've delayed dealing with the problem so far. I'll update this topic after I try one of the suggested methods. I'm leaning towards zeroing the file. – objectoriented Mar 6 '12 at 18:01

It may be faster to zero/truncate the file than remove it. I also mention this because that's a really large log file, so there must be a tremendous amount of process activity writing to it. Try : > /path/to/logfile.log if you're not in a position to stop and start the production services.

| improve this answer | |
  • Thanks! I was able to stop the process writing to the file so no additional data is being appended to the file. – objectoriented Dec 2 '11 at 15:58
  • Go ahead and zero it. – ewwhite Dec 2 '11 at 16:25
  • Great idea, thank you! Took absolutely ZERO io when truncating a 2 GB file, which normally halts the server for 20 seconds. – Shane N Mar 23 '15 at 21:29
  • Truncating a file to zero bytes need to perform the same amount of work to free the blocks as deleting the file does. But truncating many times removing a few MB each time can spread the load over a long time if sleeps are inserted between each truncate. – kasperd Apr 28 '15 at 7:28

ionice -c3 rm yourfile.log is your best shot, then rm will belong to idle I/O class and only uses I/O when any other process does not need it. ext3 is not stellar when deleting huge files and there's not very much you can do about it. Yes, the rm command will slow down your system. The amount of slowness and the duration of the deletion is something one can only guess, it depends so much on hardware, kernel version and ext3 file system creation settings.

For log servers and other servers with large files I tend to use XFS, as it is very fast with them.

| improve this answer | |
  • 1
    Thanks Janne. I worry there won't much idle time? I think ionice -c2 -n7 rm big.log is more appropriate. – objectoriented Dec 2 '11 at 6:17
  • I have no idea if your server has lots of free disk I/O time or not. But which one of you is more important to you: 1) time it takes to remove the file or 2) your server stays ~smooth during deletion? – Janne Pikkarainen Dec 2 '11 at 7:06
  • Method with least disruption is the goal. My point here is that thing is a web server (LAMP to be specific) so it's business is IO. I just don't see the disks sitting idle very much. I worry a giant file removal would take days or longer if the system is waiting for periods of zero IO. – objectoriented Dec 2 '11 at 15:53
  • IONICE is the kicker here. I spent hours trying to figure out what was slowing down my server while deleting a substantial amount of files. "find . -type f -delete -print" was using 99.9% of my I/O and other processes were stacking up because they could not complete their I/O. – Nick Woodhams Aug 15 '12 at 11:53

Alternate solution is having separate disks and cycle between them. So when your done logging to one disk, you swap to the other, and then you could use lots of IO to remove stuff, and not burden the active disk.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.