We are currently using syslog-ng to dump files to a network storage location. Each day there are 5 .log files that are written by various servers and at the end of the day I need to merge the 5 files in chronological order and then compress them. For the past 2 years, I've use logmerge and it's worked great. The exact syntax is:

/local/bin/logmerge -f /mnt/logs/Windows/`date -d yesterday +\%Y-\%m-\%d`-sys*.log | gzip -9 -c > /mnt/logs/Windows/`date -d yesterday +\%Y-\%m-\%d`.log.gz && rm -f /mnt/logs/Windows/`date -d yesterday +\%Y-\%m-\%d`-sys*.log

Over the past few weeks this process has broken due to how large the .log files have gotten. Each one is now over 7 GB and the logmerge process is failing on sorting so many lines. Right now I'm just gzipping them but it makes searching harder because the logs aren't in order.

Is there a better way to merge these files and zip them up?

  • I know this doesn't exactly answer your question, but why don't you use a central syslog server which writes the concatenated file? – Lacek Mar 23 '17 at 14:48
  • So we have 5 syslog-ng servers that are writing these log files. They are all front ended by an F5 for load balancing. The storage location I'm writing to doesn't allow writes of 1 file from 5 different servers unfortunately. So each one has to write it's own file and then I merge them after the day is over. – Eric Mar 23 '17 at 14:53
  • If you have a computer nearby (a management node, perhaps), you could use it as a central log server. The others would forward the events to the central server, so it could write the log file. This way, the concatenated log file would be written from one node only. – Lacek Mar 23 '17 at 15:00
  • 2
    Are the 5 logs each already sorted, because if so you can greatly speed up the final sort (seen in your logmerge link) by a sort -m i.e. a merge not a full sort. Also, are all your log events on one line, and not split over several lines, as logmerge is putting some effort into handling this case, and perhaps for you it isnt needed. – meuh Mar 23 '17 at 15:36
  • If you forward to 1 server, it adds double the network overhead to forwarding all those events and then you kill the HA concept of relying on just 1 server. As far as using -m, each of the 5 files is in chrono order but if I use the -m, it would be chrono order for 5 different files, not 1 large file in full chrono order. If you search for abc it would search abc of file 1 and show 9:00 am and then maybe abc of file 2 further down and may be 5:00 am. – Eric Mar 23 '17 at 19:32

It rather sounds like you may want to look into some form of database to store your logs.

One possibility might be to use the ELK stack:

  • Elasticsearch as the database (it is based on Lucene, so is geared towards searching, but also provides a number of aggregation, map-reduce, and related functionality)
  • Logstash as your log ingestion and parser agent - you can amongst other things use the syslog input to receive logs from your nodes (you can send them either directly, or use your local syslog-ng daemon to feed a copy to logstash)
  • Kibana is used to visualise, search, and manipulate your logs.

It isn't necessarily the answer you might have been looking for, but it sounds like you might have a legitimate use case for a solution like it. You can also consider something like Splunk, but given your volume of data, that will cost you.

Logstash can also be used on Windows machines to read the EventLog, so might allow you to achieve your goals without using syslog at all (if I am reading between the lines of your setup correctly).

It may also be there is something you can do about how the logs are being written to help avoid such massive files, but I would tend to think that if you are regularly dealing with 7GB of logs you periodically need to search through, a solution geared towards that use case might be more practical.

Updated I see. In which case, is it not possible to have syslog-ng write everything either to one massive daily file (rather than 5), or to have syslog-ng write everything to a series of files up to a certain size (e.g. 10 700M files, each created after the last fills)?

It really sounds like the issue is having your data out of order, and I would have thought there are ways to avoid that issue, by configuring syslog accordingly. Since it sounds like the timestamps are more important than the sources, I would imagine that timestamps alone (or possibly, timestamps and maximum log size) should determine how events are stored in the first place.

  • We already have a 10 node ELK cluster where we store logs for ~2 weeks for a more "pretty" search for our end users. The storage I'm talking about is raw text storage and we want it for a lot of random scripts we utilize and for 6+ months of archiving. – Eric Mar 23 '17 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.