Problem with Logstash and Opendistro Elasticsearch

Hi, community. I tried to write some data to OpenDistro Elasticsearch using logstash.
This is my Logstash output config:

output {
    if "_dateparsefailure" not in [tags] and [@metadata][kafka][topic]=="squid_before_logstash" {
        elasticsearch {
            hosts => "hzzp://<es_node>:9200"
            index => "logstash_test_index_%{+YYYY.MM.dd}"
            user => admin
            password => admin
        }
    }
    else if  "_dateparsefailure" in [tags] and [@metadata][kafka][topic]=="squid_before_logstash" {
        file {
            path => "/opt/logstash_test/dateparsefailure-%{+YYYY-MM-dd}.txt"
        }
    }
}

But I got those errors in Logstash log:

[2021-06-18T05:39:18,719][ERROR][logstash.outputs.elasticsearch][main] Unable to get license information {:url=>"hzzp://user:password@<elastic_ip>:9200/", :exception=>LogStash::Outputs::ElasticSearch::hzzpClient::Pool::BadRespons
eCodeError, :message=>"Got response code '400' contacting Elasticsearch at URL 'hzzp://<elastic_ip>:9200/_license'"}
[2021-06-18T05:39:18,720][ERROR][logstash.outputs.elasticsearch][main] Could not connect to a compatible version of Elasticsearch {:url=>"hzzp://user:password@<elastic_ip>:9200/"}
[2021-06-18T05:39:21,094][ERROR][logstash.outputs.elasticsearch][main][81dcded0d793f2d1ce54996d7b2accca6703353fa04380b0df10cf210dbf9aa0] Elasticsearch setup did not complete normally, please review previously logged errors {:message=>"No
 Available connections", :exception=>LogStash::Outputs::ElasticSearch::hzzpClient::Pool::NoConnectionAvailableError}
[2021-06-18T05:39:23,722][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"hzzp://user:password@<elastic_ip>:9200/"}

versions:
logstash-1:7.13.0-1
opendistroforelasticsearch-1.13.2-1, Elasticsearch 7.10.2

Also I tried make this settings at logstash.yml

xpack.monitoring.enabled: false
xpack.management.enabled: false

What I am doing wrong? Logstash and Elastic must have same version?
I replaced http with hzzp, because Forum says me, that I can’t put more than 2 link in my message

Elastic and Logstash version must be same

Hey @ThreatInter - this is a known thing. Take a look at the announcement about Logstash 7.13

@searchymcsearchface

to me it looks like that elastic have but a license check in version 7.13 of the OSS *beats / logstash effectively breaking interoperability with open search.

Looking at how opensearch came about I could see this as a move from elastic to have their OSS versions of *beats / logstash work with their OSS version of elastic - only.

what is your position on this? are you going to fork and maintain the *beats and logstash to in order to ensure support?

here is the pull that i believe is the “move” from elastic
https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1005
not being able to keep software updated as possible vunlerabilities arise etc, will in my view keep orgs from using open search, as they will have policies / requirements in place that requires them to keep software updated.

Hey @ssi0202. That’s known and people have been working on it for quite some time. Expect a release of a output plugin for Logstash that works with OpenSearch shortly (possibly today!). In the mean time, you can take a look at the source and build it yourself on github GitHub - opensearch-project/logstash-output-opensearch: A Logstash plugin that sends event data to a OpenSearch clusters and stores as an index.

Beats is more complicated - the way beats is put together doesn’t allow for a more graceful solution without drastic moves. However, this seems to only really be an issue when you are using Beats directly connected to OpenSearch, which is possible but not a very typical setup. If you’re using Beats → Logstash → OpenSearch, you should be fine after the output plugin.

hi @searchymcsearchface thanks for the quick reply. i will try it out when the plugin is done sounds like its just around the cornor.

so it would still be OSS *beats → OSS logstash (with the plugin) ->opensearch correct ?

im not sure i agree with the beats → opensearch is not that widely used, based on my experience, but you most likely have better data on that :wink:

Yup. Disclaimer: I’ve not personally used OSS beats → logstash → opensearch, but this is what I’ve been told should work. It’s on my list to write up (look for that soon).

If this is a greenfield project, I’d look at something like Fluentbit instead of Beats. My personal POV is that Fluentbit has a bright future.

WRT beats->directly to OpenSearch, I’d like to know more! Do you or someone you know use this setup?

Is there any news related to the problem from the beginning?

The logstash oss version was released a few days ago and we tested it within our testing lab, based on all opensearch components.

We also get the following error message, within logstash:

logstash               | [2021-08-16T11:37:26,979][ERROR][logstash.outputs.elasticsearch][main] Unable to get license information {:url=>"https://admin:xxxxxx@xx.xx.xx.xx:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '400' contacting Elasticsearch at URL 'https://xx.xx.xx.xx:9200/_license'"}
logstash               | [2021-08-16T11:37:26,979][ERROR][logstash.outputs.elasticsearch][main] Could not connect to a compatible version of Elasticsearch {:url=>"https://admin:xxxxxx@xx.xx.xx.xx:9200/"}

Which logstash version should be used with opensearch?

Also opened an bug report [BUG] Elasticsearch output fail - Could not connect to a compatible version of Elasticsearch · Issue #54 · opensearch-project/logstash-output-opensearch · GitHub

@thsul According to the documentation, 7.13.x

Thank you. Logstash is working now, the wrong plugin was used. Details can be found in the “issue” on github.

1 Like

Hi @searchymcsearchface
I tried to downgrade the Logstash version to 7.13.4 OSS build and still no luck connecting to Opendistro Elasticsearch (1.11.0).
Here’s my logstash config:

input {
    beats {
        port => 5000
        codec => "json_lines"
    }
}

output {
    elasticsearch {
        ilm_enabled => false
        user => "admin"
        password => "PASSWORD"
        ssl => true
        ssl_certificate_verification => false
        hosts => "https://127.0.0.1:9200"
        index => "test-%{+YYYY.MM.dd}"
        cacert => "/path/to/root-ca.pem"
       }
}

I tried this previously on Logstash OSS 7.14.0 and same error.

Error:

[2021-08-19T03:47:05,246][ERROR][logstash.outputs.elasticsearch][main] Could not connect to a compatible version of Elasticsearch {:url=>"https://admin:xxxxxx@127.0.0.1:9200/"}
[2021-08-19T03:47:10,253][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://admin:xxxxxx@127.0.0.1:9200/"}
[2021-08-19T03:47:10,265][ERROR][logstash.outputs.elasticsearch][main] Unable to get license information {:url=>"https://admin:xxxxxx@127.0.0.1:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '500' contacting Elasticsearch at URL 'https://127.0.0.1:9200/_license'"}
[2021-08-19T03:47:10,265][ERROR][logstash.outputs.elasticsearch][main] Could not connect to a compatible version of Elasticsearch {:url=>"https://admin:xxxxxx@127.0.0.1:9200/"}

Do you have the OpenSearch Output plugin? You’ll need that for any version of either OSS Elasticsearch or OpenSearch.

Nope. I did not have.
However, I installed the plugin and switched config from “elasticsearch” to “opensearch” in the output section and things started to work fine! I also removed the “ilm_enabled” field.
Thanks :slight_smile:

2 Likes