2

We have multiple file servers where I work containing terabytes and terabytes worth of files. We have no system in place currently for archiving/deleting stale data. My task is to create a script that will move/delete files older than 3 years. Moving/deleting the files aside, just "getting" the files is an issue. Using Powershell so far.

I'm running into files where the filename/path is longer than ~248 characters, which Powershell doesn't seem to be able to handle. I've also looked into Robocopy, but Robocopy has no way to access the LastAccessTime file property. Plus Powershell seems to be really slow. Here is the code I have been testing with:

Get-ChildItem "\\path" -Force -Recurse | 
        where {!($_.PSIsContainer) -and ($_.LastAccessTime -lt (Get-Date).AddYears(-3))} | 
        select Mode,LastAccessTime,Length,Name

My question is this: is there a faster, more efficient way to "get" terabytes worth of files recursively (including the LastAccessTime property) where some paths are longer than 248 chars?

Looking for free solutions mostly, but if there are some good paid solutions, I would be willing to check them out.

4 Answers 4

1

If you're on Win2k8 or newer, you can use File Server Resource Manager to create a File Expiration task which will move files older than x to a different directory.

1

The code example will be slow, as you're enumerating every single file and directory recursively, before applying the filter. This is not PowerShell's fault, the underlying filesystem classes in .Net are actually very fast.

You want to use the builtin filter parameters instead of piping to where-object (in general, do filtering as far "left" as possible.)

2
  • +1, he is actually running 2 commands instead of 1, not sure if that's doubling the time it takes to execute or squaring it.
    – MDMoore313
    Commented Dec 16, 2013 at 18:20
  • I'm not using the code example I posted because of long paths. Commented Dec 16, 2013 at 18:48
0

I got bit by powershell not long ago whilst doing something similar. It takes ages and ages. Luckily I had Cygwin installed and could test the corresponding maneuvre using native *nix commands. The difference in time taken was staggering.

So speedwise the slowness has less to do with windows or the filesystem per se, but more to do with the execution efficiency of the language used.

Path depth/length I have not yet experimented with, but for these very time consuming file handling tasks I now use cygwin over powershell on a windows machine. It sucks, but there it is.

There are probably other, more recommendable alternatives which also make for speedier file operations than the windows native commands. If there's one there's many is my thinking on that.

3
  • Tried Cygwin, but it was incredibly slow for me. Commented Dec 16, 2013 at 18:46
  • I'm sorry to hear that, it is obviously not the solution suited to your task. I'm filing your experience under very interesting though, as I use it next to daily just for performance reasons. Is it possible to briefly describe the test you did?
    – ErikE
    Commented Dec 16, 2013 at 19:24
  • I typed find "./pathtofolder" in Cygwin. Using System.IO.DirectoryInfo in PowerShell, it took about 2.8 min to finish the command. However, Cygwin took 14 minutes. Robocopy takes less than 2 minutes. Same folder for all. Commented Dec 16, 2013 at 20:51
0

I am just going to use Robocopy and go by LastModifiedDate.

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .