The Agents and ingestion tools site says:
The override main response setting compatibility.override_main_response_version is deprecated from OpenSearch version 1.x and removed from OpenSearch 2.0.0. This setting is no longer supported for compatibility with legacy clients.
And the compatibility matrix for beats doesn’t have a row for OpenSearch v2.0. In my current testing filebeat OSS 7.10.2 will not work with opensearch 2.0 (attempting to connect to Elasticsearch version 2.0.0).
Is there no longer a way to use the legacy beats with opensearch as of v2.0.0?
Thanks.
To be more specific, where it fails now is:
2022-06-01T14:10:08.825Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 2.0.0
2022-06-01T14:10:08.876Z INFO template/load.go:117 Try loading template filebeat-7.10.2 to Elasticsearch
2022-06-01T14:10:10.596Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(http://opensearch:9200)): Connection marked as failed because the onConnect callback failed: error loading template: could not load template. Elasticsearch returned: couldn't load template: 400 Bad Request: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:
...
Then it dumps out the template. In my case I’m trying to use OSS filebeat 7.10.2 with the nginx module. It worked with OpenSearch v1.3.x with the compatibility.override_main_response_version setting.
Disabling loading templates:
2022-06-01T14:19:03.008Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(http://opensearch:9200)): Connection marked as failed because the onConnect callback failed: 1 error: Error loading pipeline for fileset nginx/access: failed to adapt pipeline for ECS compatibility: user_agent processor requires option 'ecs: true', but Elasticsearch {2.0.0 2 0 0 } does not support this option (Elasticsearch 6.7.0 or newer is required)
Thank you @tlacuache for reporting the issue. I’ll have the team review on our side.
Thanks! In the meantime I think I’ll look at having the beats output to Logstash OSS and then to opensearch via the Logstash OpenSearch output plugin rather than OS directly as I think that’s another path forward.
Quick update.
The team has a compatibility model fix in the works which you can track. Having said that, pointing agents to an aggregator ahead of pushing it to OpenSearch is the right model as it reduces the amount of total connections pointing directly at OpenSearch.
Thank you for raising the concern @tlacuache. Much appreciated.
Another update @tlacuache, the fix for Beats compatibility mode was deployed yesterday.