Best Practices for OpenSearch 3

General Feedback
,
1

Hello everyone,

We are currently using OpenSearch 2.19.3 running on RHEL 8 virtual machines, with Opensearch deployed inside Podman containers. We use own Opensearch builds with UBI base. Our data ingestion pipeline is handled by Logstash. We are about to update on Opensearch 3.

We use this logstash: opensearchproject/logstash-oss-with-opensearch-output-plugin - Docker Image

Our Setup:

  • We ship documents from Kubernetes using Beats: auditbeat, filebeat, heartbeat, and metricbeat.

  • These Beats feed into Logstash, which also processes logs from some Dockerized applications.

  • Additionally, several RHEL 8 VMs with applications send logs directly to Logstash.

  • We rely on Logstash because we need multiple inputs (beats, TCP), as well as processing filters like grok, ECS compatibility, mutate, and enrichment.

  • Our ingest:

image
image376×379 27.2 KB

Use Case:

  • Store enriched logs for 3 months.

  • Perform aggregations on the data for up to 2 years.

  • Dashboards are built on keywords and numeric data.

  • Alerting is configured based on document counts and difference between buckets.

Questions:

  1. Are there any recommendations or best practices to improve this architecture? Something new that we should try in Opensearch 3?

  2. Is Logstash still fully supported as an output for OpenSearch 3?

  3. Are there any updated or official Docker images build on UBI for OpenSearch 3 that you would recommend?

Thanks in advance for your advice!

2

I think Logstash 9 should be supported to OpenSearch 3.x . I haven’t fully tested everything but my Initial test showed it works once you install the logstash-output-opensearch plugin and set xpack.monitoring.enabled: false and pipeline.ecs_compatibility: disabled.

we’re using logstash 7.x for OpenSearch 3.2 without issue and in progress of moving to Logstash 9.x

1 Like
3

What is the UBI base in We use own Opensearch builds with UBI base.

Are you referring GitHub - opensearch-project/user-behavior-insights: User Behavior Insights plugin for OpenSearch?