Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am working on setting up a syslog-ng server to accept logs from various sources. The general setup is going to be:

Device1
Device2                               Syslog1
Device3   -->   Load balancer   -->   Syslog2   -->   Shared SAN storage
Device4                               Syslog3
Device5

I am using 514/udp and with this setup, I should easily have HA and be able to round robin and spread the load. My concern is how the storage is going to be shared and written to on the backend. Ideally I'd like each device to have a deviceX.log file for the day. What happens if syslog1 gets a large log file in from device1 and is still writing to device1.log when syslog2 gets a small log file in from device1 and starts trying to write to the same file at the same time?

I have read that in Linux if you append the file and don't open it in "write" mode, then you will never get the write protected error. I am not sure how syslog does this.

So my questions are:

  1. What shared file system would you recommend for the SAN to share across the syslog boxes?
  2. Is there a better way to do the multiple syslog servers to 1 file? If syslog just appends, then I should be fine.

Any recommendations would be greatly appreciated. Thanks in advance.

share|improve this question
add comment

2 Answers

Not sure what you are using for a load balancer, but I would recommend that on your load balancer you configure some sort of "stickiness" so that Device1 logs will always get directed to Syslog1, etc.

This would address your concern about the log file, but it would also prevent what I would consider to be a problem that is just as bad.

My concern with your example would be if there were a lot of log entries passed from Device1 to Syslog1 and then a smaller number passed from Device1 to Syslog2 is that Syslog2 might be able to insert the later syslog messages out of order into your log file.

share|improve this answer
    
I fully agree. Depending on the load balancer, I'm going to try to have it maintain some sort of connection based biase for when it hits 1 server, it will stay there unless it goes down. The out of order logs are bad, but not horrible as they will all be time stamped and searchable. In an ideal world I would obvious rather them be in direct order. –  Eric May 10 '13 at 18:17
add comment
up vote 0 down vote accepted

I found that any sort of cluster file system such as GlusterFS would work fine for multiple servers writing to one file. However due to the SAN infrastructure I have, that was not possible. Therefore I ended up writing a file per day per syslog cluster.

Syslog1 --> 2012-07-29-sys01.log
Syslog2 --> 2012-07-29-sys02.log

I then used the logmerge script and a cronjob to combine the files and sort them by the correct time. The cron job below runs the day after the files are done being written to.

/local/bin/logmerge -f /logs/date -d yesterday +\%Y-\%m-\%d-sys*.log | gzip -9 -c > /logs/date -d yesterday +\%Y-\%m-\%d.log.gz && rm -f /logs/date -d yesterday +\%Y-\%m-\%d-sys*.log

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.