Versions* (relevant - OpenSearch/Dashboard/Server OS/Browser):
Describe the issue:
I am sending logs from fluent bit to msk kafka to logstash to opensearch. My msk kafka has three brokers. But the brokers keeps on getting full even after configuring the retention policy to one day.
Configuration:
Fluent bit → msk kafka → logstash → opensearch
Relevant Logs or Screenshots:
Hey @vamsi.ramisetti
Normally when I see “time outs” its a configrution or connection issue. That been said, How did you configure Kafka to send messages to Logstash and how did you ocnfigure Logstash to receive thoses messages?
I have built a cluster in msk with m5.2xlarge and i have assigned three brokers. So in my fluent bit configuration in the output i have defined kafka configuration and in the input of logstash i have defined my kafka brokers. In this way i have configured kafka
Hey @vamsi.ramisetti
By chance, are you using certificates and/or credentials for the connection between msk kafka && Logstash?