Recomendations for k8s logs shipping operator

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

2.3.0

Describe the issue:

Looking for recommendations for Kubernetes logging operator to collect logs from all pods and k8s hosts misc logs. The one we use now runs into performance issues where logs collection stops .

Configuration:

Current setup fluentbit → fluentd → kafka <----> logstash → Opensearch.

Relevant Logs or Screenshots:

Hi @tru64jurus,

The setup you have looks pretty good and it may need some tuning on k8s and fluentbit side.
I would like to know more about the performance issues you have.

Thank you,
Nicoale Vartolomei

Elasticsearch/OpenSearch & Solr Consulting, Production Support & Training Sematext Cloud - Full Stack Observability

@Nicolaegis Here is example of issues reported fluentbit not recovering after forward upstream fluentd restart .

This logging operator will enable you to push logs into OpenSearch the same way your current configuration works:

fluentbit → fluentd → <----> logstash → Opensearch.
Logging → Flow → output (OpenSearch)

@mikeyGlitz this logging operator have an issue with fluentbit failing to reconnect to fluentd after fluentd statefulset restart .