9

I have a huge log file I need to delete on a production web server. I'm worried it'll bring the system to a crawl if I rm it on Linux. Any brilliant ideas?

Update:

Filesystem: ext3

Partition: /var (mostly logs and MySQL data)

Log file is no longer being written to. (No additional data is being appended)

Web Server is LAMP (lots of IO)

4
  • 1
    Duplicate of serverfault.com/questions/128012/… ? What is your underlying file system?
    – Ben DeMott
    Dec 2, 2011 at 5:55
  • ext3 - hoping for something more inspiring that ionice might work... I read the man page (linux.die.net/man/1/ionice) and it sounds like the right tool... I'd like a someone with experience using it to chime in. I don't feel very adventurous when it comes to production servers. Dec 2, 2011 at 6:06
  • Did any of these solve the problem?
    – ewwhite
    Mar 5, 2012 at 2:14
  • No, I've delayed dealing with the problem so far. I'll update this topic after I try one of the suggested methods. I'm leaning towards zeroing the file. Mar 6, 2012 at 18:01

3 Answers 3

12

It may be faster to zero/truncate the file than remove it. I also mention this because that's a really large log file, so there must be a tremendous amount of process activity writing to it. Try : > /path/to/logfile.log if you're not in a position to stop and start the production services.

4
  • Thanks! I was able to stop the process writing to the file so no additional data is being appended to the file. Dec 2, 2011 at 15:58
  • Go ahead and zero it.
    – ewwhite
    Dec 2, 2011 at 16:25
  • Great idea, thank you! Took absolutely ZERO io when truncating a 2 GB file, which normally halts the server for 20 seconds.
    – Shane N
    Mar 23, 2015 at 21:29
  • Truncating a file to zero bytes need to perform the same amount of work to free the blocks as deleting the file does. But truncating many times removing a few MB each time can spread the load over a long time if sleeps are inserted between each truncate.
    – kasperd
    Apr 28, 2015 at 7:28
10

ionice -c3 rm yourfile.log is your best shot, then rm will belong to idle I/O class and only uses I/O when any other process does not need it. ext3 is not stellar when deleting huge files and there's not very much you can do about it. Yes, the rm command will slow down your system. The amount of slowness and the duration of the deletion is something one can only guess, it depends so much on hardware, kernel version and ext3 file system creation settings.

For log servers and other servers with large files I tend to use XFS, as it is very fast with them.

4
  • 1
    Thanks Janne. I worry there won't much idle time? I think ionice -c2 -n7 rm big.log is more appropriate. Dec 2, 2011 at 6:17
  • I have no idea if your server has lots of free disk I/O time or not. But which one of you is more important to you: 1) time it takes to remove the file or 2) your server stays ~smooth during deletion? Dec 2, 2011 at 7:06
  • Method with least disruption is the goal. My point here is that thing is a web server (LAMP to be specific) so it's business is IO. I just don't see the disks sitting idle very much. I worry a giant file removal would take days or longer if the system is waiting for periods of zero IO. Dec 2, 2011 at 15:53
  • IONICE is the kicker here. I spent hours trying to figure out what was slowing down my server while deleting a substantial amount of files. "find . -type f -delete -print" was using 99.9% of my I/O and other processes were stacking up because they could not complete their I/O. Aug 15, 2012 at 11:53
1

Alternate solution is having separate disks and cycle between them. So when your done logging to one disk, you swap to the other, and then you could use lots of IO to remove stuff, and not burden the active disk.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.