3

I have a server with a huge file system that is used for continues writes and reads without hiccups and needs to be active at all times.

My goal is to have a cronjob that deletes empty folders and files based on name and older than X amount of days. The obvious answer is to use find -name -mtime -delete but it uses too much disk resources and causing IO wait and other programs to miss their checks because of server unresponsiveness.

Is there any easy to use tool that does this kind of work? maybe limit disk usage? something else i haven't thought of?

  • Maybe this question points in the right direction? – bjoster May 7 '19 at 9:07
  • What it the type of the filesystem? – Mark Wagner May 8 '19 at 15:50
5

Use ionice to reduce the priority of your find command and minimize the impact on other applications.

ionice -c 3 find -name -mtime -delete

-c 3 is class "idle" and a program running with idle io priority will only get disk time when no other program has asked for disk IO (for a defined grace period). The impact of idle IO processes on normal system activity should be zero.

If that never happens (likely on a system that sees use 24x7) try for instance -c 2 -n 7 for the lowest best-effort priority

2

Most file systems do not deal with changes to millions of files efficiently, they do a tremendous amount of metadata I/O. Hopefully you have less than 100,000 files per directory.

Throttling with io priorities and/or cgroups can make the system responsive, but may cause this cleanup to not complete in a reasonable time.

On Linux LVM, you can add a caching layer with lvmcache. Add the fastest SSD you have so it can service several thousand IOPS. Throttling is probably still needed, if you require your I/O response times to not degrade.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.