Hi,
in the documentation is somehow not clear whether the opensearch in long-term support filebeat OSS.
Currently opensearch supports Filebeat OSS 7.12.1. Does it mean the opensearch not support Beat OSS 7.12.2 at all? We don’t want to use logstash, that means we need to migrate from filebeat OSS to other agent like fluentbit in the future?
Thanks for clarifying this
I hate to say it, but I believe you’re correct. Elastic doesn’t seem to have any intention of allowing their product to output to OpenSearch, although I’m happy to be corrected there. Some alternative workflow will likely have to be made. I was about to mention Logstash… but…
Sorry to hear you don’t want to use Logstash - was there some piece of your workflow that makes it undesirable? I’m curious about what your situation is that makes it unappealing. Maybe there’s something we can help you with?
Fluentbit is a very well supported and mature product as well - you should be able to accomplish your goals with it just as well. Some assembly will be required. 
Nate
Hi @nateynate,
thanks for your reply. About your question, we have a very huge workload and we had a big delay during the ingesting the logs. We tried to tune up logstash by scaling it and adding a layer caching before logstash, but it didn’t help in long-term. That is why we don’t use it anymore.
For fluentbit we did some POC and we have faced some loadbalancing issue on our coordinating nodes. That means logs are ingesting only to one coordinating node and it caused high cpu usage, whereas on the other node was no workload loadbalanced. It is something which we should investigate further. We run the opensearch on on k8s.
Thanks again for clarification
Amir
Hey @amirkh81, on the last section of the page you linked it mentions that anything in the OSS 7.12 line should work 7.12.2 included so long as they are following semantic versioning correctly.
Nate is right though, there have been some breaking changes past 7.12 that render us incompatible unfortunately. Have you opened any issues on fluentbit’s repos regarding your issues?
Hi @dtaivpp ,
Yes you are right. I’ve mentioned the version 7.12.2 only as example to indicate the newer version of filebeat OSS. It could be version 7.13.* or something newer. We want to decide in long-term for choosing logging agent and unfortunately we should switch to other options due the incompatibility issue of the newer filebeat OSS version with opensearch.
Regarding the fluentbit, actually still no. It is still ongoing.
Thanks
Amir
Interesting that you recommend logstash as the solution. Currently logstash and OS are in a pretty bad location. In that OS will stop accepting messages from logstash every hour without intervention. This was broken in OS 2.0 and is still in need of a fix. Pretty much preventing anyone that uses logstash to upgrade to OS 2.0 or higher. Or those that have to do some hacky scripts to keep their logs flowing.
The corresponding issues. [BUG] java.io.OptionalDataException error in OpenSearch 2.0 rc · Issue #1961 · opensearch-project/security · GitHub and https://github.com/opensearch-project/security/issues/1927
Yeah, that bug is a bummer. It will be fixed for sure - was just trying to discover if there was something else besides a current bug. Sans regressions like the above, it’s a pretty good solution. Something it couldn’t do or something that made it an inappropriate solution other than a painful temporary regression. 
Let us know if there’s more we can do for you!
Yeah it is a deal breaker for me to upgrade to 2.x. Hopefully it gets fixed soon. Upgrading to 2.x would be really nice.
The nice thing is it seems like there has been a lot of good conversation around this and I just noticed this draft PR opened literally 29 minutes ago. Here’s to hoping it’s a functioning fix 