2

I have a standard ELK stack currently storing numerous log outputs. I'm trying to separate my indices to be source-specific.

As part of my FileBeats config, some standard fields are always generated as part of every message (and are location-specific), which I want to use as the basis for my my ES index:

output {
  elasticsearch {
    hosts => ["http://elasticsearch.mydomain.com:80"]
    index => "logstash-%{+YYYY.MM.dd}-%{[fields][identifier]}"
  }
}

However, ES is rejecting some of the indices as the field contains uppercase characters - identifier has acceptable values like myfoo but also could be MyBar:

[logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {"reason"=>"Invalid index name [logstash-2017.06.02-MyBar], must be lowercase"}

The casing isn't essential and I can add a mutate filter to forcibly lowercase the fields in question, but I would prefer to store the identifier field with proper casing, yet use the lower-cased version for the index name.

Is there function which can be called in the elasticsearch output, to lower-case the field in question? Something like

    index => "logstash-%{+YYYY.MM.dd}-%{[fields][identifier]}.lowercase()"
2

This can be done with a bit more mutate trickery.

  1. Create a new field using mutate, set to your identifier.
  2. In a second mutate, lowercase that new field.
  3. Use the new field in your output.

Like so:

filter {
  mutate {
    add_field => { "lc_identifier" => "%{fields}%{identifier}" }
  }
  mutate {
    lowercase => [ "lc_identifier" ]
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch.example.com:80"]
    index => "logstash-%{+YYYY.MM.dd}-%{[lc_identifier]}"
  }
}

You will end up with an lc_identifier field in your indices, but that shouldn't matter much.

0

Building on sysadmin1138's answer:

You can use the @metadata field to prevent the lc_identifier field from appearing in your documents while indexed.

In your filter, change:

filter {
  mutate {
    add_field => { "lc_identifier" => "%{fields}%{identifier}" }
  }
  mutate {
    lowercase => [ "lc_identifier" ]
  }
}

To:

filter {
  mutate {
    add_field => { "[@metadata][lc_identifier]" => "%{fields}%{identifier}" }
  }
  mutate {
    lowercase => [ "[@metadata][lc_identifier]" ]
  }
}

And to call it in your output:

output {
  elasticsearch {
    hosts => ["http://elasticsearch.example.com:80"]
    index => "logstash-%{+YYYY.MM.dd}-%{[@metadata][lc_identifier]}"
  }
}

This way you can create an index pattern with a lowercase identifier, but avoid having the lowercase field appear in the documents themselves.

-1
filter {
  mutate {
    add_field => { "lc_identifier" => "%{fields}" }
  }
  mutate {
    lowercase => [ "lc_identifier" ]
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch.example.com:80"]
    index => "logstash-%{+YYYY.MM.dd}-%{lc_identifier}"
  }
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.