3

We have a challenge in deleting circa X million files that meet a certain criteria - specifically must be over 90 days old and exclude certain file formats. To date, we've been using a multi-threaded powershell script which is good but still not as fast as we need as our ingest processes are creating new files at a high rate.

What I can do to accelerate the deletion of so many files?

closed as off-topic by Tero Kilkanen, Daniel, Jacob Evans, yagmoth555, EEAA Jan 18 '17 at 21:45

This question appears to be off-topic. The users who voted to close gave this specific reason:

If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    Software recommendations are off-topic. Maybe you can ask in another way? But can I ask two stupid questions? What kind of process creates millions of files? And why doesn't it take care of deletion? – Daniel Jan 18 '17 at 12:25
  • Oh. I didn't realise that. Can I ask which of the Stack Exchange forums is the appropriate forum then? Thanks! :) To answer the question the process is the nature of the work and it's a bit complex but we can ingest something in the region of a few million 1kb small files a week. – K-Lye Jan 18 '17 at 12:46
  • 1
    There is Software Recommendations. – Gerald Schneider Jan 18 '17 at 12:51
1

Agent Ransack is really fast, and supports deleting files based on modified date.

Also, obviously, it will run faster if you shut down all the other processes running on the server.

0

Powershell has a reputation of not being very fast; maybe a batch file using Windows build-in forfiles command can do the trick.

To select and delete all files older than 90 days, you can use something similar to:

forfiles -p "C:\your\dir" /S /D -90 /C "cmd /c del @path"

If this remains too slow, you can use forfiles to fist create a list of the to-be-deleted files, and than use a batch remove command over all files in the list (this can significantly decrease the number of del calls/iterations).

For more informations and other more or less creative mode to delete files, you can see here and here

  • Thanks but I was looking for something a bit more off-the-shelf and I've asked this question i the wrong forum so will close the question. – K-Lye Jan 18 '17 at 12:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.