Describe the issue:
I wanna use the newest version of Opensearch - version 3.1.
At the moment I’m using Opensearch v2.19 and filebeat is sending the logs directly to Opensearch.
Is this also possible with Opensearch v3.1 ? Which version of filebeat is supporting this ?
Or do I have to send the logs by filebeat via logstash to Opensearch ?
Which versions of filebeat and logstash has to be used in this case ?
I know the compatibility matrix, but it’s not so easy to understand.
Next step is ingesting logs from k8s cluster to Opensearch. I assume this is the same setup.
@Detlef Elastic has introduced a version check in Filebeat 7.13. It won’t allow you to connect with OpenSearch.
filebeat-oss_7.13.0 | 2021-07-27T12:47:34.618Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://node-0.example.com:9200)): Connection marked as failed because the onConnect callback failed: could not connect to a compatible version of Elasticsearch: 400 Bad Request: {"error":{"root_cause":[{"type":"invalid_index_name_exception","reason":"Invalid index name [_license], must not start with '_'.","index_uuid":"_na_","index":"_license"}],"type":"invalid_index_name_exception","reason":"Invalid index name [_license], must not start with '_'.","index_uuid":"_na_","index":"_license"},"status":400}
filebeat-oss_7.13.0 | 2021-07-27T12:47:34.618Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://node-0.example.com:9200)) with 2 reconnect attempt(s)
Try using Logstash with the OpenSearch output plugin and newer Filebeat as a workaround.