Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I know it sounds odd but I need a slower or cached filesystem.

I have a lot of firewalls that are syslog'ing their data to a pair of Linux VMs which write these files to their 'local' (actually FC SAN attached) ext3-formatted disks and also forward the messages to our Splunk servers.

The problem is that the syslog server is writing these syslog messages as hundreds, sometimes thousands, of tiny ~4k writes per second back to our FC SAN - which can handle this workload right now but our FW traffic's going to be growing by at least a factor of 5000% (really) in coming months and that'll be a pain for the SAN, I want to fix the root cause before it's a problem.

So I need some help figuring out a way of getting these writes cached or held-off in some way from the 'physical' disks so that the VMs fire off larger, but less frequent, writes - there's no way of avoiding these writes but there's no need for it to do so many tiny ones.

I've looked at the various ext3 options, setting noatime and nodiratime but that's not made much of a dent in the problem. Obviously I'm investigating other file systems but thought I'd throw this out in case others have the same problem in the future.

Oh and I can't just forward these messages to Splunk, our firewall team insist they're in their original format for diag purposes.

share|improve this question
    
Why don't you use splunk forwarder? –  rvs Nov 26 '11 at 5:29
    
We do already but we need to keep the original FW-only flat file logs for our security guys. –  Chopper3 Nov 26 '11 at 8:34

2 Answers 2

Perhaps commit ext3 mount option would help you? For example, commit=60 would flush all the data and the meta-data only once per minute.

Obligatory warning: this might lead to data loss of up to one minute of data (if you pass the commit=60 value).

share|improve this answer
    
Isn't that just for journal commit, not data? –  Chopper3 Nov 23 '11 at 10:08
    
That's what I originally thought, but mount man page says about commit Sync all data and metadata every nrsec seconds. The default value is 5 seconds. Zero means default. ... perhaps together with commit and data=journal you could achieve what you want? With data=journal All data is committed into the journal prior to being written into the main file system. –  Janne Pikkarainen Nov 23 '11 at 10:43
    
It might be the data=journal bit I need, thanks so far. –  Chopper3 Nov 23 '11 at 10:55

Filesystem: Disable write barriers if your devices are using them, and disable atime updates across the board.

But chances are you can also tune your syslog at the expense of greater loss of data in the case of a failure event (power, etc).

Example directives from syslog-ng (which may not be what you are using):

  • flush_lines() Specifies how many lines are flushed to a destination at a time. Syslog-ng waits for this number of lines to accumulate and sends them off in a single batch.

  • flush_timeout() Specifies the time syslog-ng waits for lines to accumulate in its output buffer. See the flush_lines option for more information.

The destination is in this case is a disk file.

share|improve this answer
    
I'd already mentioned I've disabled atime and we're not using syslog-ng either - I'll have a look at the write barriers though. –  Chopper3 Nov 25 '11 at 21:05
    
Which syslog server are you using? If it's vanilla syslogd, you can prefix the destination filename with a dash - to prevent every line from being forced to disk. –  MikeyB Nov 25 '11 at 21:31
    
If the problem is significant enough, would you consider changing to a syslog daemon which does support what you need to do? –  evilensky Nov 25 '11 at 22:09
    
Not sure how much choice I've got in that area, our security guys tend to get what they want –  Chopper3 Nov 26 '11 at 8:35

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.