0

Currently I have the following setup.

Syslog --> Logstash --> ElasticSearch --> Kibana

Logstash is creating a daily index "/etc/elasticsearch/data/test-elasticsearch/nodes/0/indices/logstash-2014.02.04" and I'm viewing all of the logs through Kibana. We want to set up some user based access control using the kibana-authentication-proxy setup due to it supporting Per-user kibana index supported. now you can use index kibana-int-userA for user A and kibana-int-userB for user B I'd like to make it where all logs coming in from logstash with a location of "/var/log/UNIX/*.log" get sent to a new index of unix-2014.02.04 instead of the logstash one. That way I can use the Kibana auth proxy to give my UNIX users access only to their logs. I've read a little about creating the mappings but wasn't sure how to tie it all together. I saw you could do various things with API calls but was curious if I could set all of this up in the elasticsearch.yml file from the start.

Thanks,

Eric

2

I found out that you can do this in the logstash configuration using input and output filters. The new if way is shown below but I haven't gotten it to work yet.

input {
  file { 
    type => "unixlogs"
    path => "/var/log/UNIX/*.log"
  } 
}

output {
  if [type] == "unixlogs" {
    elasticsearch { 
      host => "localhost"
      index => "unix-%{+YYYY.MM.dd}"
    }
  }
}

Below is the older way to do it that I have gotten to work.

file {
    type => "syslog"
    exclude => ["*.gz"]
    start_position => "end"
    path => [ "/var/logs/Security/*.log"]
  }

file {
    type => "unix-syslog"
    exclude => ["*.gz"]
    start_position => "end"
    path => [ "/var/logs/UNIX/*.log"]
  }

output {
    elasticsearch {
    type => "unix-syslog"
    embedded => "false"
    host => "X.X.X"
    cluster => "my-elasticsearch"
    index => "unix-%{+YYYY.MM.dd}"
  }
    elasticsearch {
    embedded => "false"
    host => "X.X.X"
    cluster => "my-elasticsearch"
 }
}

With the top way it will only write to the one index you tell it to. With the bottom way, the UNIX logs will write to the unix index and the generic index since it's not if, just and.

  • The elastic guys plan to get rid of type attribute and it also won't apply if you already have type field in your source (it doesn't override). So I advise using tags or add_field instead. – BornToCode Jun 21 '17 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.