It rather sounds like you may want to look into some form of database to store your logs.
One possibility might be to use the ELK stack:
- Elasticsearch as the database (it is based on Lucene, so is geared towards searching, but also provides a number of aggregation, map-reduce, and related functionality)
- Logstash as your log ingestion and parser agent - you can amongst other things use the syslog input to receive logs from your nodes (you can send them either directly, or use your local syslog-ng daemon to feed a copy to logstash)
- Kibana is used to visualise, search, and manipulate your logs.
It isn't necessarily the answer you might have been looking for, but it sounds like you might have a legitimate use case for a solution like it.
You can also consider something like Splunk, but given your volume of data, that will cost you.
Logstash can also be used on Windows machines to read the EventLog, so might allow you to achieve your goals without using syslog at all (if I am reading between the lines of your setup correctly).
It may also be there is something you can do about how the logs are being written to help avoid such massive files, but I would tend to think that if you are regularly dealing with 7GB of logs you periodically need to search through, a solution geared towards that use case might be more practical.
Updated I see. In which case, is it not possible to have syslog-ng write everything either to one massive daily file (rather than 5), or to have syslog-ng write everything to a series of files up to a certain size (e.g. 10 700M files, each created after the last fills)?
It really sounds like the issue is having your data out of order, and I would have thought there are ways to avoid that issue, by configuring syslog accordingly.
Since it sounds like the timestamps are more important than the sources, I would imagine that timestamps alone (or possibly, timestamps and maximum log size) should determine how events are stored in the first place.