Monitoring solution that doesn't duplicate source data?

Question

I need to analyze data stored in multiple databases. Each of them contains a few TB of logs and sensor data.

If I use Splunk or ElasticSearch/Kibana, I see 2 solutions:

1. Batch import everything

2. Write some scripts to import only the data I need, on demand

This is not optimal because with 1. I duplicate TB of data, and with 2. I need some custom logic that won't be integrated with my analytics tool.

My questions are:

• Is there a way to perform analytics in place with Splunk or Kibana, i.e. skip the importing step and just read everything from the existing databases?

• Or is there a tool that would automatically fetch the relevant data and only the relevant data?

Answer 1

To be able to search you data quickly i advice Elasticsearch Depending on what the logic behind relevant data should be you can use ElasticSearch Rivers http://www.elasticsearch.org/guide/en/elasticsearch/rivers/current/

This allows Elastic search to pull all (relevant) data. My experience says the TB of data you have stored will be reduced to several GB in ElasticSearch