Fluent bit is conceptually a little different - you have a pipeline of inputs, processors, and filters where as with the beats family, you just have the beat module. To support those data sources get the right input and define the rest of the pipeline.
There is probably more details in the fluent bit community about how to handle those other data sources through the other parts of the pipeline.
So, after this long discussion, we have analyzed all the aspects based on our product and decided to go with elastic beats with fluentD. So overall workflow will be given below -
elastic beats â kafka â fluentD â opensearch.
The main reason for choosing elastic beats is the flexibility of creating a custom module that beats provide over a fluent bit, hence based on this decision and community support we have decided to go with beats family as a data shipper.
And, choosing between logstash and fluentD, it seems fluentD provides lots of plugins and many out-of-the-box features which makes fluentD the right choice.
Whats your thoughts on this decision?
You should be ok, especially since you are decoupling Beats from anything else.
My only thoughts are that Beats might not be super active in the future. Plus when Beats blocked OpenSearch in the past, you have to question if they decide they donât like Kakfa the same could happen, and itâs out of your hands. Shippers are sticky, so carefully consider if you think the future is good as far as development and openness.
Ok. Thanks for sharing your views.
@searchymcsearchface @amitai I didnât find any resources which explain how to configure fluentD to send output data to OpenSearch, can you provide me the references to do that?
Yep. This is a temporary situation. Try this:
https://github.com/uken/fluent-plugin-elasticsearch/issues/915#issuecomment-933351153
Within the next n weeks weâll the opensearch-ruby client will be out and there will likely be a better solution.
Ok understood. Is there any recommandation/guidelines whether itâs safe to use fluent-plugin-elasticsearch with opensearch? This seems to be workaround. Also can I get some approx estimates how long this n weeks are?
I wouldnât worry about using it too much. OpenSearch is binary compatible with ES 7.10.2 and people have been using the fluentd with ES together for years. The changes are to the elasticsearch-ruby client and that is seemingly motivated by the fork. By pegging to the previous version of the library youâre getting around those issues.
Timeline: Well, the opensearch ruby plugin needs to come out first - that should be GA in a few weeks, then the new library needs to be pushed to fluent-plugin-elasticsearch after. The OpenSearch team has less control over that, but itâs a simpler task.
Thanks, It worked.
I am looking into different features supported under elastic X-Pack, what is the vision of X-Pack in opensearch?
X-Pack in OpenSearch? Probably never going to happen. X-Pack isnât open source. AFAIK X-Pack is also going away in ES as they are going for a different distribution model since they eliminated open source ES earlier this year.
Now, there are features that overlap between OpenSearch and X-Pack already. Indeed, Open Distro for Elasticsearch developed a series of open source plugins to fill the feature gap that X-Pack created (honestly, this is reductive, but you get the idea). Those plugins evolved into being part of what OpenSearch is today.
Anything in particular from X-Pack that youâre interested in?
I see. Can you provide a reference link where it mentions the features that OpenSearch overlap with x-pack? along with that the feature currently I am in interest with is security analytics, threat detection, and detection rules.
So far I found one project in the current road map - Support security analytics - Threat detection, intrusion detection ¡ Issue #109 ¡ opensearch-project/alerting ¡ GitHub.
Can you provide me with the list of features or references that are going to integrate with this project?
Humm. I donât think there is a matrix because the features arenât always 1:1. Perhaps you can zoom on what you want to do with those categories and maybe I can point you in the right direction?
Yeah, Thanks for your answers.
So the very first feature is -
Elastic have so many predefined detection rules which have some risk score and itâs mapped with the Mitre Tactics.
I wanted to know how risk score is being calculated and how we will achieve this in opensearch?
