1

I am working on setting up a syslog-ng server to accept logs from various sources. The general setup is going to be:

Device1
Device2                               Syslog1
Device3   -->   Load balancer   -->   Syslog2   -->   Shared SAN storage
Device4                               Syslog3
Device5

I am using 514/udp and with this setup, I should easily have HA and be able to round robin and spread the load. My concern is how the storage is going to be shared and written to on the backend. Ideally I'd like each device to have a deviceX.log file for the day. What happens if syslog1 gets a large log file in from device1 and is still writing to device1.log when syslog2 gets a small log file in from device1 and starts trying to write to the same file at the same time?

I have read that in Linux if you append the file and don't open it in "write" mode, then you will never get the write protected error. I am not sure how syslog does this.

So my questions are:

  1. What shared file system would you recommend for the SAN to share across the syslog boxes?
  2. Is there a better way to do the multiple syslog servers to 1 file? If syslog just appends, then I should be fine.

Any recommendations would be greatly appreciated. Thanks in advance.

2 Answers 2

0

Not sure what you are using for a load balancer, but I would recommend that on your load balancer you configure some sort of "stickiness" so that Device1 logs will always get directed to Syslog1, etc.

This would address your concern about the log file, but it would also prevent what I would consider to be a problem that is just as bad.

My concern with your example would be if there were a lot of log entries passed from Device1 to Syslog1 and then a smaller number passed from Device1 to Syslog2 is that Syslog2 might be able to insert the later syslog messages out of order into your log file.

1
  • I fully agree. Depending on the load balancer, I'm going to try to have it maintain some sort of connection based biase for when it hits 1 server, it will stay there unless it goes down. The out of order logs are bad, but not horrible as they will all be time stamped and searchable. In an ideal world I would obvious rather them be in direct order.
    – Eric
    May 10, 2013 at 18:17
0

I found that any sort of cluster file system such as GlusterFS would work fine for multiple servers writing to one file. However due to the SAN infrastructure I have, that was not possible. Therefore I ended up writing a file per day per syslog cluster.

Syslog1 --> 2012-07-29-sys01.log
Syslog2 --> 2012-07-29-sys02.log

I then used the logmerge script and a cronjob to combine the files and sort them by the correct time. The cron job below runs the day after the files are done being written to.

/local/bin/logmerge -f /logs/date -d yesterday +\%Y-\%m-\%d-sys*.log | gzip -9 -c > /logs/date -d yesterday +\%Y-\%m-\%d.log.gz && rm -f /logs/date -d yesterday +\%Y-\%m-\%d-sys*.log

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .