Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstas

General Feedback
1

Hey people,

I’m trying to get rid of this message which is flooding my CloudWatch. My setup is as follows:

...
ContainerDefinitions:
        - Essential: true
          Image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2
          Command:
            - "-e"
            - !Sub |
                input {
                   tcp {
                     port => 9601
                     codec => json
                   }

                   udp {
                     port => 9602
                     codec => json
                   }

                   sqs {
                     queue => "${LogstashFallbackQueue.QueueName}"
                     id_field => "sqs_id"
                     polling_frequency => 20
                     region => "${AWS::Region}"
                     threads => 1
                  }
                }
                filter {
                  mutate {
                    remove_field => ["path", "host", "port"]
                  }
                }
                output {
                  opensearch {
                    index  => "logstash-%{+YYYY.MM.dd}"
                    hosts => ["${OpensearchEndpoint}"]
                    auth_type => {
                      type => "aws_iam"
                      region => "${AWS::Region}"
                    }
                  }
                }
          Name: logstash
...

I have been trying to find information on what I need to change to get rid of this message, but most search results seem to be from Elasticsearch and not this specific plugin for Opensearch.

2

Hi @deleugpn,

What version of Logstash are you using? Logstash OSS 8.0 introduces a breaking change where all plugins run in ECS compatibility mode by default. If you use a compatible OSS client you must override the default value to maintain legacy behavior:

ecs_compatibility => disabled

Please let us know if you still run into the issue once making the adjustment to your Logstash config.

3

I have the same warning message. I have logstash version 7.16.3.
I set the ecs_compatibility = disabled on pipeline level, it helped to reduce some warnings but still, some warnings are there. I have tried to set ecs_compatibility => disabled in pipelines as well. but no luck.

I have tried this as well, but no luck

Additionally, we see the following error, it seems it tries to look for 2x.json template and that does not exist. Do we need to install something special?

[2022-06-27T12:23:17,971][ERROR][logstash.outputs.opensearch][my_pipeline] Failed to install template {:message=>"Failed to load default template for OpenSearch v2 with ECS disabled; caused by: #<ArgumentError: Template file '/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/templates/ecs-disabled/2x.json' could not be found>", :exception=>RuntimeError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/template_manager.rb:33:in load_default_template’", “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/template_manager.rb:21:in install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:412:in install_template’”, “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:247:in finish_register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:224:in block in register’”, "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/plugin_mixins/opensearch/common.rb:83:in block in after_successful_connection'"]}