Logstash bottleneck

OpenSearch
1

Hi, anyone have an idea for better configuration for logstash? We are getting a lot of kafka lags

image (22)
image (22)664×586 28.7 KB

2

For consuming data from Kafka by Logstash, there’re some best practices:

  1. Group the Logstash instances, make sure that a big Kafka topic is consumed by a single group of Logstash instances, and other small or medium topics are consumed by other groups.
  2. Make sure that the partition numbers of a topic equals to the number of consumer_threads in Logstash.
    , hope this helps.