OpenSearch only receives logs once per hour

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Logstash - 7.17.11
Logstash OpenSearch Output - 1.2.0
OpenSearch - Helm Chart Version 2.9.0

Describe the issue:
Currently we have Elasticsearch set up, but we are moving to OpenSearch. I’ve set up OpenSearch output following the guide, but OpenSearch is only receiving logs once per hour, while Elasticsearch receives them almost constantly.

The Logstash logs don’t seem out of the ordinary, so I am not sure if this is an issue with the OpenSearch Output setup, or some OpenSearch config I missed.

Configuration:
Logstash Output:

output {
  if "event-timed-out" in [tags] {
    elasticsearch {
      id => "fallback-elasticsearch-output"
      hosts => ["localhost:9200"]
      index => "fallback-bisappslogs-%{+YYYY.MM.dd}"
    }
    opensearch {
        index => "fallback-bisappslogs-%{+YYYY.MM.dd}"
        hosts => ["host"]
        user  => "${}"
        password => "${}"
        ssl_certificate_verification => true
    }
  } else {
    elasticsearch {
      id => "elasticsearch-output"
      hosts => ["localhost:9200"]
      index => "bisappslogs-%{+YYYY.MM.dd}"
    }
    opensearch {
        index => "bisappslogs-%{+YYYY.MM.dd}"
        hosts => ["host"]
        user  => "${}"
        password => "${}"
        ssl_certificate_verification => true
    }
  }
}

Relevant Logs or Screenshots:

2

Do you use Enterprise certificates in OpenSearch?

3

The k8s cluster OpenSearch is deployed too has a certificate for the OpenSearch ingress. Though I would have thought any issues with this would have stopped all logs, and given a connection error?

4

@Ziggiyzoo That’s true. You should get a connection error straight away when starting the Logstash.

Have you tried using only the opensearch output without the if statement?

5

@pablo Using only OpenSearch output without the if statement has increased the amount and regularity of logs being received.

I’ll see if I can find out if that if statement is still necessary, and will get back to you if I have any more issues, thanks! :slight_smile: