Logstash-oss with elasticsearch filter plugin connecting to opensearch

General Feedback
1

Hi,

We’ve been using logstash-oss 7.12.1 with an elasticsearch filter section which looked like this:

filter {
 if [agent][type] == "winlogbeat" and [event][code] == 4625 {
        elasticsearch {
              hosts => "https://elasticsearch.random.svc.cluster.local:9200"
              index => "filebeat-*"
              user => "admin"
              password => "password"
              query_template => "/usr/share/logstash/config/filebeat-admin-lookup.json"
              fields => { "Group" => "Group" }
              ca_file => "/usr/share/logstash/config/root-ca.pem"
              tag_on_failure => ["first_elasticsearch_lookup_failure"]
              id => "filter_elasticsearch_lookup_first_endpoint"
           }
      }
}

Because of log4shell we had to update logstash-oss container to version 7.16.1 with opensearch output bundled. However, after this we cannot execute our pipeline because we get this error:

[2021-12-17T15:17:16,883][ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>“main”, :exception=>#<LogStash::ConfigurationError: Could not connect to a compatible version of Elasticsearch>, :backtrace=>[“/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:247:in `block in healthcheck!'”, “org/jruby/RubyHash.java:1415:in `each’”, “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:240:in `healthcheck!'”, “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:374:in `update_urls’”, “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:89:in `update_initial_urls’”, “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:83:in `start’”, “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:359:in `build_pool’”, “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:63:in `initialize’”, “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in `create_http_client’”, “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in `build’”, “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:34:in `build_client’”, “/usr/share/logstash/vendor/bund
le/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch.rb:275:in `register’”, “org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register’”, “org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register’”, “/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins’”, “org/jruby/RubyArray.java:1821:in `each’”, “/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins’”, “/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:589:in `maybe_setup_out_plugins’”, “/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `start_workers’”, “/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in `run’”, “/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start’”], “pipeline.sources”=>[“/usr/share/logstash/pipeline/logstash.conf”], :thread=>“#<Thread:0x4dd2c7fc run>”}

Main part being this: “Could not connect to a compatible version of Elasticsearch”.

Is there a work-around for this? We tried with the workaround here:

PUT _cluster/settings
{
  "persistent": {
    "compatibility": {
      "override_main_response_version": true
    }
  }
}

But we still get the same error.

Thanks

2

Hi.

Upstream removed the ability to write to ElasticSearch < 7.11 from logstash-output-elasticsearch since plugin version 11.0.0 (shipped with Logstash 7.13.0).

Maybe try the logstash-output-opensearch plugin? Caveat, I’ve not tried it against an ES cluster.

1 Like
3

You can fetch Logstash 7.16.1 plus the logstast-output-opensearch-plugin bundled together from the OpenSearch downloads page here: Opensearch 2.2.1 · OpenSearch

4

Thank you both, however i am not talking about the elasticsearch/opensearch output, i am talking about the filter plugin for elasticsearch which we use for lookups in other indices before we index events.

Elasticsearch filter plugin | Logstash Reference [7.16] | Elastic

5

Looks like upstream removed the ability to query in elasticsearch-ruby 7.14.0 shipping with logstash-filter-elasticsearch 3.11.0 in logstash 7.16.

I’m not aware of a replacement filter plugin for OpenSearch. This may also be the first request for it.

The case for the output plugin was raised here and one was made. Consider submitting a similar request for the filter plugin?

1 Like
6

Thank you for this information. I’ve created a feature request here: Logstash Elasticsearch Filter for OpenSearch · Issue #4 · opensearch-project/opensearch-clients · GitHub