Best way to ingest the live log stream data to open search

Hi All,

Requirement :

Can anyone please suggest me the best way to ingest the live log stream data (which was running and generating the new logs always) to OpenSearch.… Finally want to see the Dashboard/Visualization with different unique fields in OpenSearchDashboard…

I have found the below combinations can be used but need your help to choose the best…

  1. Logstash
  2. FileBeat + logstash
  3. FileBeat + Kafka +logstash
  4. Kafka + Fluentd

If possible please refer me to some best document to choose it…

Thanks In Advance…


Generally, there is no one-sized-fits-all approach, so I can only comment on higher-level issues.

The beat family (filebeat, etc.) should be considered a bit of a gamble in context of OpensSearch. I’m not personally sure of the future of that software (the creators have left Elastic and there newer product that fills the same nice) and there has been actions to box OpenSearch out of using in some contexts.

Logstash, like beats, has seen some actions to prevent connection to OpenSearch. However, there is a Logstash OpenSearch Output plugin that mitigates any connection issues, so I think it’s a safer bet than Beats. Fluentd is is used by a lot of folks and fills the same niche as Logstash. It’s more independent, so I wouldn’t expect to ever see any unresolvable issues with OpenSearch.

However, both Fluentd and Logstash are written in Ruby, a fine language but one not know for performance nor efficiency and that’s been a sticking point for a lot of folks.

Here is some context on OpenSearch with Logstash and Beats

Sorry to spam some older posts - but wanted to mention for fluentd - please look to the newly released fluent-plugin-opensearch
link: GitHub - fluent/fluent-plugin-opensearch: OpenSearch Plugin for Fluentd

1 Like