ElasticSearch Multiple Indexes and Routing

Question

Currently I have the following setup.

Syslog --> Logstash --> ElasticSearch --> Kibana

Logstash is creating a daily index "/etc/elasticsearch/data/test-elasticsearch/nodes/0/indices/logstash-2014.02.04" and I'm viewing all of the logs through Kibana. We want to set up some user based access control using the kibana-authentication-proxy setup due to it supporting Per-user kibana index supported. now you can use index kibana-int-userA for user A and kibana-int-userB for user B I'd like to make it where all logs coming in from logstash with a location of "/var/log/UNIX/*.log" get sent to a new index of unix-2014.02.04 instead of the logstash one. That way I can use the Kibana auth proxy to give my UNIX users access only to their logs. I've read a little about creating the mappings but wasn't sure how to tie it all together. I saw you could do various things with API calls but was curious if I could set all of this up in the elasticsearch.yml file from the start.

Thanks,

Eric

Answer 1

I found out that you can do this in the logstash configuration using input and output filters. The new if way is shown below but I haven't gotten it to work yet.

input {
  file { 
    type => "unixlogs"
    path => "/var/log/UNIX/*.log"
  } 
}

output {
  if [type] == "unixlogs" {
    elasticsearch { 
      host => "localhost"
      index => "unix-%{+YYYY.MM.dd}"
    }
  }
}

Below is the older way to do it that I have gotten to work.

file {
    type => "syslog"
    exclude => ["*.gz"]
    start_position => "end"
    path => [ "/var/logs/Security/*.log"]
  }

file {
    type => "unix-syslog"
    exclude => ["*.gz"]
    start_position => "end"
    path => [ "/var/logs/UNIX/*.log"]
  }

output {
    elasticsearch {
    type => "unix-syslog"
    embedded => "false"
    host => "X.X.X"
    cluster => "my-elasticsearch"
    index => "unix-%{+YYYY.MM.dd}"
  }
    elasticsearch {
    embedded => "false"
    host => "X.X.X"
    cluster => "my-elasticsearch"
 }
}

With the top way it will only write to the one index you tell it to. With the bottom way, the UNIX logs will write to the unix index and the generic index since it's not if, just and.