I need to analyze data stored in multiple databases. Each of them contains a few TB of logs and sensor data.
If I use Splunk or ElasticSearch/Kibana, I see 2 solutions:
Batch import everything
Write some scripts to import only the data I need, on demand
This is not optimal because with 1. I duplicate TB of data, and with 2. I need some custom logic that won't be integrated with my analytics tool.
My questions are:
Is there a way to perform analytics in place with Splunk or Kibana, i.e. skip the importing step and just read everything from the existing databases?
Or is there a tool that would automatically fetch the relevant data and only the relevant data?