We hope to use ElasticSearch Input Plugin from Logstash to output with ElasticSearch.

This feature has been successfully collected by MongoDB Input Plugin in real time.

However, the ElasticSearch Input Plugin does not start with the data that you recently collected when you run the collection. Duplicate data is created when collecting all data.

I would like to collect data from my most recent collection. How do I set it up?

There is another question.

The Mongo input plugin distinguishes recently collected data by "placeholder_db_dir", "placeholder_db_name" settings.

What is the ElasticSearch input plugin?

I am config file info.

input {
  elasticsearch {
    hosts => ''
    index => 'sample'
    scroll => '5m'
    docinfo => true

filter {
        json {
                source => 'message'

output {
        elasticsearch {
                hosts => ["localhost:9200"]
                index => "es"
        stdout { codec => rubydebug }
        file {
                path => '/home/jskang/jskang-platform/logs/logstash/logstash-%{+YYYY.MM.dd}.log'
                codec => rubydebug

If your intent is to capture changes to logstash in the last N minutes, you will need to manage your query and scheduling more directly. The elasticsearch input does not have the concept of a since_db, so you will need to do it through use of a structured query => and use a schedule =>. Such as...

query    => [your ES query, returning everything in the last 2 minutes]
schedule => "/2 * * * *"

This will run the input collection every 2 minutes, and return everything with a timestamp in the last 2 minutes.

  • And I ask one more question. your talking result is logstash does not support incremental collection. Did I get it right? – junsung kang Dec 26 '18 at 9:15
  • @junsungkang Yes, the ElasticSearch input does not support incremental collection. – sysadmin1138 Dec 29 '18 at 1:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.