We have log data (XML data) going into a folder from specific machines. So for example the log data goes into a flat folder structure like so.
This folder can have between 50-300k files at any given time. Once a file is put into this folder it doesn't get updated. The files have some information about what was logged (information about what/where it was processed) and the yyyymmddhhmmss.xml appended to the end of it.
The files that go into this are kept around between 3-6 weeks and then eventually get purged back down (this is typically done by some other process that I have no control over).
The challenge is to archive several years of this data. I am planning on creating a folder structure of
\\ArchiveServer\Machine1\logs\YYYY\mm\DD\*.xml. This will allow us to get to the data on a certain day easier (if needed) and where we won't end up having 1M+ files in a single directory.
I am trying to figure out a good way to manage this sync/merge between the old structure and the new one continuously where it will scale enough to handle the number of files we will eventually have without slowing down too much.
In Python I was doing a listing of the
\\machine1\logs\*.xml source directory grabbing that file list, and then doing a recursive directory listing to get the list of all files in the
\\ArchiveServer\Machine1\logs destination, compare the source list and the destination list and if there are any files from the source list not in the destination list I copy them over to the destination (archive) folder.
Unfortunately this has the problem of having to do a directory listing on the
\\Archiveserver which will eventually get slow as more data is copied into the archive folder. For only 200k files it takes about 30 seconds a machine to do a listing on the destination folder. I am concerned that once it gets to 500k-1M records it will take significantly longer.
Is there a better way of accomplishing this that will scale with the number of files I will be dealing with (this is in Windows)?