We have log data (XML data) going into a folder from specific machines. So for example the log data goes into a flat folder structure like so.

\\Machine1\logs\*.xml

This folder can have between 50-300k files at any given time. Once a file is put into this folder it doesn't get updated. The files have some information about what was logged (information about what/where it was processed) and the yyyymmddhhmmss.xml appended to the end of it.

The files that go into this are kept around between 3-6 weeks and then eventually get purged back down (this is typically done by some other process that I have no control over).

The challenge is to archive several years of this data. I am planning on creating a folder structure of \\ArchiveServer\Machine1\logs\YYYY\mm\DD\*.xml. This will allow us to get to the data on a certain day easier (if needed) and where we won't end up having 1M+ files in a single directory.

I am trying to figure out a good way to manage this sync/merge between the old structure and the new one continuously where it will scale enough to handle the number of files we will eventually have without slowing down too much.

In Python I was doing a listing of the \\machine1\logs\*.xml source directory grabbing that file list, and then doing a recursive directory listing to get the list of all files in the \\ArchiveServer\Machine1\logs destination, compare the source list and the destination list and if there are any files from the source list not in the destination list I copy them over to the destination (archive) folder.

Unfortunately this has the problem of having to do a directory listing on the \\Archiveserver which will eventually get slow as more data is copied into the archive folder. For only 200k files it takes about 30 seconds a machine to do a listing on the destination folder. I am concerned that once it gets to 500k-1M records it will take significantly longer.

Is there a better way of accomplishing this that will scale with the number of files I will be dealing with (this is in Windows)?

  • 1
    I am concerned that once it gets to 500k-1M records it will take significantly longer. It almost certainly will. Patient: "Doctor, it hurts when I do this." Doctor: "Well, don't do that." Is there a better way? Yes. Don't stuff a million files into one directory. That's a bad design and it won't scale. – Andrew Henle Sep 19 '17 at 10:17
  • @AndrewHenle Do you mean a single folder with 500k-1M records? I don't have a folder with that. Most of the leaf directories have no more than 5k files in the directory. So if I have a million files I would have ~200 folders with no more than 5000 files in each folder. Not just a single folder with a million files. The issue is on how to manage even that going forward, as more files get generated, the more files I have to get a listing from which will eventually take more time. Which is why I'm asking for questions on a better approach. – Kevin Vasko Sep 19 '17 at 15:11

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.