Best practice for ingesting log messages from K8s to Opensearch

DevOps
1

Hi,

what’s best practice for ingesting application log messages from K8s to a separate Opensearch cluster ?

All application logs of different K8s cluster should be sent to a Opensearch Cluster.

The Opensearch Cluster is a separate cluster.
The newest version of Opensearch should be used.

Which ingestion tools are recommended ?
Filebeat does not work with the newest Opensearch version. So IMHO the logstash should be used between filebeat and opensearch. Is this correct ?
How can I install logstash fail-safe ? Should run logstash inside K8s ? Should it run on every K8s node together with filebeat ?
Are there other ingestion tools which are recommended for my use case ?

Thnx in advance !
Detlef

2

@Detlef Any Filbeat below 7.14 will work with OpenSearch. Elastic introduced version check for most of its products in 7.14.

You can use newer versions of the Filebeat only with Logstash. For Logstash 7.14+ you must use opensearch as output with logstash-output-opensearch plugin installed.

Alternatively you might consider using Data Prepper instead of Logstash.

1 Like
3

@pablo
So I can use following pipeline ?

filebeat 9.x → logstash 7.13.x-7.16.x with opensearch output plugin-> opensearch 3.x

4

@Detlef According to Logstash compatibility matrix, Lostash 7.x won’t support Fileabt 9.x. You must match the major versions.

According to the last version commit of the OpenSearch output plugin, the latest supported version of Logstash is 8.15.3

5

@Detlef I’ve just ran a Docker container with the following Dockerfile and version 9.2.0. The plugin has been installed successfully and Logstash worked.
I would suggest testing the Logstash and the output plugin functionality before putting into production.

ARG LOGSTASH_VER

FROM docker.elastic.co/logstash/logstash-oss:${LOGSTASH_VER}

RUN logstash-plugin install logstash-output-opensearch

/usr/share/logstash/bin/logstash-plugin list --verbose 
...
logstash-output-opensearch (2.0.3)
...
6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.