Logstash-oss with opensearch plugin compatibility issues

OpenSearch Client Libraries
1

Hello,
In my lab I have an Opensearch node and I want to process data with Logstash-oss-with-opensearch. For that reason I am using the Logstash Elasticsearch Input plugin (NOT output, I know there is a specific opensearch output plugin).

Since I am running latest Opensearch, 1.2.4, I assumed the latest Logstash-OSS with opensearch plugin 7.16.2 (from the same page) would work. Unfortunately running the 7.16.2, I get:

[2022-02-08T10:09:52,398][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Could not connect to a compatible version of Elasticsearch>, :backtrace=>["/home/dcvythoulkas/workspace/logstash/logstash-oss-7.16.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.12.1/lib/logstash/inputs/elasticsearch.rb:481:in `test_connection!'", "/home/dcvythoulkas/workspace/logstash/logstash-oss-7.16.2/vendor/bundle/jruby/2.5.0/gems/logstash-input-elasticsearch-4.12.1/lib/logstash/inputs/elasticsearch.rb:243:in `register'", "/home/dcvythoulkas/workspace/logstash/logstash-oss-7.16.2/vendor/bundle/jruby/2.5.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'", "/home/dcvythoulkas/workspace/logstash/logstash-oss-7.16.2/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/home/dcvythoulkas/workspace/logstash/logstash-oss-7.16.2/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins'", "/home/dcvythoulkas/workspace/logstash/logstash-oss-7.16.2/logstash-core/lib/logstash/java_pipeline.rb:390:in `start_inputs'", "/home/dcvythoulkas/workspace/logstash/logstash-oss-7.16.2/logstash-core/lib/logstash/java_pipeline.rb:315:in `start_workers'", "/home/dcvythoulkas/workspace/logstash/logstash-oss-7.16.2/logstash-core/lib/logstash/java_pipeline.rb:189:in `run'", "/home/dcvythoulkas/workspace/logstash/logstash-oss-7.16.2/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["/home/dcvythoulkas/workspace/logstash/elk-lab-export.conf"], :thread=>"#<Thread:0x66024e6d run>"}

Versions up to 7.13.2 work with no issues. Is this incompatibility because of this or is it something else I am missing?

Thanks,
dcv

2

Isn’t the opensearch plugin specifically an output plugin? You say you are using the elasticsearch input plugin, connecting to a opensearch 1.2.4 instance. I’m not sure if that will work unfortunately. Have you tried setting this?

PUT _cluster/settings
{
  "persistent": {
    "compatibility": {
      "override_main_response_version": true
    }
  }
}
3

Isn’t the opensearch plugin specifically an output plugin?
Yes it is, and afaik there is no opensearch input plugin, I only mentioned to emphasize that I have trouble with the elasticsearch plugin that has no equivalent so far.
You say you are using the elasticsearch input plugin, connecting to a opensearch 1.2.4 instance. I’m not sure if that will work unfortunately.

I think the same, just making sure according to the documentation.

I also tried your suggestion but it didn’t change anything,

4

Upstream added a product check in elasticsearch-ruby 7.14.0 which made its way into logstash-input-elasticsearch in v4.12.0. The input plugin was upgraded to v14.12.1 in Logstash 7.16. Logstash versions prior to 7.16 shouldn’t have this check.

I’m not aware of a replacement input plugin for OpenSearch. This may also be the first request for it.

The case for the output plugin was raised here and one was made. Consider submitting a similar request for the input plugin?

5

@shdubsh we are actively working on logstash input plugin for OpenSearch. Will keep you posted.

1 Like
6

@vamshin Glad to hear it!

I was unable to find the issue in Github. Please share if it is available to the community. Thanks!

7

@shdubsh sorry for not sharing the link. Please find below link for the issue.

https://github.com/opensearch-project/opensearch-plugins/issues/121

2 Likes