OSS Logstash 7.16.1 incompatible with Open Distro Elasticsearch 1.13.3

Open Source Elasticsearch and Kibana
,
1

I am running an upgrade from some older versions of opendistro and OSS logstash to the latest versions, in order to mitigate some of the latest vulnerabillities.

According to Elastic, Logstash OSS 7.16.x should be compatible with Elasticsearch 7.10.x .

And according to OpenSearch/OpenDistro/AWS (Cool cats have many names), Open Distro Elasticsearch 1.13.3 should be running Elasticsearch 7.10.2 under the hood.

However, when I run up a cluster with Logstash-OSS 7.16.1 using an output pipeline to an Open Distro Elasticsearch 1.13.3, I’m getting the following incompatibility error in Logstash OSS:

[2021-12-17T08:12:02,583][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.

[2021-12-17T08:12:02,598][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.

[2021-12-17T08:12:02,608][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.

[2021-12-17T08:12:02,618][INFO ][logstash.outputs.elasticsearch][output-elasticsearch_local] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://elkstack-node:9200"]}

[2021-12-17T08:12:02,639][INFO ][logstash.outputs.elasticsearch][output-elasticsearch_local] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://admin:xxxxxx@elkstack-node:9200/]}}

[2021-12-17T08:12:02,717][ERROR][logstash.javapipeline    ][output-elasticsearch_local] **Pipeline error {:pipeline_id=>"output-elasticsearch_local", :exception=>#<LogStash::ConfigurationError: Could not connect to a compatible version of Elasticsearch>**, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:247:in `block in healthcheck!'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:240:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:374:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:89:in `update_initial_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:83:in `start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:359:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:63:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in `create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in `build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:34:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch.rb:275:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:589:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/outputs/output-elasticsearch_local.logstash.conf"], :thread=>"#<Thread:0x368d0eba run>"}

[2021-12-17T08:12:02,723][INFO ][logstash.javapipeline    ][output-elasticsearch_local] Pipeline terminated {"pipeline.id"=>"output-elasticsearch_local"}

[2021-12-17T08:12:02,729][ERROR][logstash.agent           ] Failed to execute action {:id=>:"output-elasticsearch_local", :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<output-elasticsearch_local>, action_result: false", :backtrace=>nil}

Output from Docker to verify versions:

someadmin@someplace:/var/git/Updated_OpenDistro_ElasticSearch/Docker$ docker ps --format "table {{.Names}}\t{{.Status}}\t{{.RunningFor}}\t{{.Image}}"
NAMES               STATUS                  CREATED             IMAGE
elkstack-kibana     Up 45 minutes       45 minutes ago      amazon/opendistro-for-elasticsearch-kibana:1.13.2
elkstack-logstash   Up 45 minutes       45 minutes ago      logstash-oss:7.16.1
elkstack-node       Up 45 minutes       45 minutes ago      amazon/opendistro-for-elasticsearch:1.13.3

My logstash output to elasticsearch pipeline:

input {
    pipeline {
        address => "output-elasticsearch_local"
    }
}

output {
      elasticsearch {
        hosts       => ["${ELASTICSERVER:not_set}"]
        ssl         => true
        cacert      => "/usr/share/logstash/config/ca.pem"
        ssl_certificate_verification => true
        user        => "${ELASTIC_LOGSTASH_USER:not_set}"
        password    => "${ELASTIC_LOGSTASH_USER_PASSWORD:not_set}"
        ilm_enabled => false
        index       => "logstash-%{[@metadata][index_prefix]}"
      }
    }

The nodes can reach each-other obviously, so the resolved variables in the above are correct. I have verified that as well, logstash is not reaching a wrong cluster, as it is an isolated environment.

Could this have to do with the indices being migrated from an old version (1.10.1)?

2

Hi.

Upstream removed the ability to write to ElasticSearch < 7.11 from logstash-output-elasticsearch since plugin version 11.0.0 (shipped with Logstash 7.13.0).

Maybe try the logstash-output-opensearch plugin? Caveat, I’ve not tried it against an ES cluster.

2 Likes
3

You can fetch Logstash 7.16.1 plus the logstast-output-opensearch-plugin bundled together from the OpenSearch downloads page here: Opensearch 1.2.2 · OpenSearch

1 Like
4

Thank you both - i blindly assumed they would work together as there was underlaying support for the Elasticsearch version, but this seems to not be the case after all.

I worked through the weekend and upgraded to Open Search instead of Open Distro, and that solved the issue.

Thank you both for your replies.

2 Likes