Automatically merge log files, keeping all lines.

Question

I have a program assigning serial numbers on a client PC (Windows XP). It writes a log to a server file share (Windows Server 2003), and also locally in case of network error. (This means that the vast majority of lines are in both files.) Every once in a while, the write to the server log does fail. Is there a good way to merge the two log files, restoring the missing entries?

I'd rather not do a full copy; there is currently some data which is only on the server copy, and I also want to leave open the possibility of multiple stations.

It should be fairly easy to do an interactive merge, but I would prefer something automatic, since it's basically set-union on the lines of the files.

Answer 1

This can be done pretty easily in PowerShell. Assuming proper local and network security, the following code can accomplish this:

$networkLog = "\\server\share\mylog.log"
$localLog = "C:\mylog.log"

# Append local log to network log
Add-Content $networkLog (Get-Content $localLog)

# Take newly appended network log, sort, filter only unique entries, save
Get-Content $networkLog | Sort-Object | Get-Unique | Out-File $networkLog

Answer 2

There's efficient no way to do it with built in commands. The easiest would be to use something like 'sort' and 'uniq' from UnxUtils:

cat log1.txt log2.txt | sort | uniq > output.txt

This basically appends the two text files, sorts them (I'm assuming each line begins with a serial number or a date), and then removes duplicate lines.

Something more efficient would be to write a small utility or script which opens both files, reads them one line at a time, and writes to the output, making sure to write each entry only once.

Answer 3

If I have this straight you have two plain text log files. You can use the windows copy command to merge the two files.

copy log1.txt+log2.txt finallog.txt

It's nice because if the log file names are static you could make a simple bat script to run the command. Even if the files are not static you can have the bat file prompt for the files you want to merge.

The problem is that the files basically just append to each other. There's really no way check for log entry dates to have the missing entries inserted into the original file using this scenario.