5

I know it sounds odd but I need a slower or cached filesystem.

I have a lot of firewalls that are syslog'ing their data to a pair of Linux VMs which write these files to their 'local' (actually FC SAN attached) ext3-formatted disks and also forward the messages to our Splunk servers.

The problem is that the syslog server is writing these syslog messages as hundreds, sometimes thousands, of tiny ~4k writes per second back to our FC SAN - which can handle this workload right now but our FW traffic's going to be growing by at least a factor of 5000% (really) in coming months and that'll be a pain for the SAN, I want to fix the root cause before it's a problem.

So I need some help figuring out a way of getting these writes cached or held-off in some way from the 'physical' disks so that the VMs fire off larger, but less frequent, writes - there's no way of avoiding these writes but there's no need for it to do so many tiny ones.

I've looked at the various ext3 options, setting noatime and nodiratime but that's not made much of a dent in the problem. Obviously I'm investigating other file systems but thought I'd throw this out in case others have the same problem in the future.

Oh and I can't just forward these messages to Splunk, our firewall team insist they're in their original format for diag purposes.

2
  • Why don't you use splunk forwarder?
    – rvs
    Nov 26, 2011 at 5:29
  • We do already but we need to keep the original FW-only flat file logs for our security guys.
    – Chopper3
    Nov 26, 2011 at 8:34

2 Answers 2

1

Perhaps commit ext3 mount option would help you? For example, commit=60 would flush all the data and the meta-data only once per minute.

Obligatory warning: this might lead to data loss of up to one minute of data (if you pass the commit=60 value).

3
  • Isn't that just for journal commit, not data?
    – Chopper3
    Nov 23, 2011 at 10:08
  • That's what I originally thought, but mount man page says about commit Sync all data and metadata every nrsec seconds. The default value is 5 seconds. Zero means default. ... perhaps together with commit and data=journal you could achieve what you want? With data=journal All data is committed into the journal prior to being written into the main file system. Nov 23, 2011 at 10:43
  • It might be the data=journal bit I need, thanks so far.
    – Chopper3
    Nov 23, 2011 at 10:55
1

Filesystem: Disable write barriers if your devices are using them, and disable atime updates across the board.

But chances are you can also tune your syslog at the expense of greater loss of data in the case of a failure event (power, etc).

Example directives from syslog-ng (which may not be what you are using):

  • flush_lines() Specifies how many lines are flushed to a destination at a time. Syslog-ng waits for this number of lines to accumulate and sends them off in a single batch.

  • flush_timeout() Specifies the time syslog-ng waits for lines to accumulate in its output buffer. See the flush_lines option for more information.

The destination is in this case is a disk file.

4
  • I'd already mentioned I've disabled atime and we're not using syslog-ng either - I'll have a look at the write barriers though.
    – Chopper3
    Nov 25, 2011 at 21:05
  • Which syslog server are you using? If it's vanilla syslogd, you can prefix the destination filename with a dash - to prevent every line from being forced to disk.
    – MikeyB
    Nov 25, 2011 at 21:31
  • If the problem is significant enough, would you consider changing to a syslog daemon which does support what you need to do? Nov 25, 2011 at 22:09
  • Not sure how much choice I've got in that area, our security guys tend to get what they want
    – Chopper3
    Nov 26, 2011 at 8:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .