Problem with Logstash and Opendistro Elasticsearch

General Feedback
1

Hi, community. I tried to write some data to OpenDistro Elasticsearch using logstash.
This is my Logstash output config:

output {
    if "_dateparsefailure" not in [tags] and [@metadata][kafka][topic]=="squid_before_logstash" {
        elasticsearch {
            hosts => "hzzp://<es_node>:9200"
            index => "logstash_test_index_%{+YYYY.MM.dd}"
            user => admin
            password => admin
        }
    }
    else if  "_dateparsefailure" in [tags] and [@metadata][kafka][topic]=="squid_before_logstash" {
        file {
            path => "/opt/logstash_test/dateparsefailure-%{+YYYY-MM-dd}.txt"
        }
    }
}

But I got those errors in Logstash log:

[2021-06-18T05:39:18,719][ERROR][logstash.outputs.elasticsearch][main] Unable to get license information {:url=>"hzzp://user:password@<elastic_ip>:9200/", :exception=>LogStash::Outputs::ElasticSearch::hzzpClient::Pool::BadRespons
eCodeError, :message=>"Got response code '400' contacting Elasticsearch at URL 'hzzp://<elastic_ip>:9200/_license'"}
[2021-06-18T05:39:18,720][ERROR][logstash.outputs.elasticsearch][main] Could not connect to a compatible version of Elasticsearch {:url=>"hzzp://user:password@<elastic_ip>:9200/"}
[2021-06-18T05:39:21,094][ERROR][logstash.outputs.elasticsearch][main][81dcded0d793f2d1ce54996d7b2accca6703353fa04380b0df10cf210dbf9aa0] Elasticsearch setup did not complete normally, please review previously logged errors {:message=>"No
 Available connections", :exception=>LogStash::Outputs::ElasticSearch::hzzpClient::Pool::NoConnectionAvailableError}
[2021-06-18T05:39:23,722][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"hzzp://user:password@<elastic_ip>:9200/"}

versions:
logstash-1:7.13.0-1
opendistroforelasticsearch-1.13.2-1, Elasticsearch 7.10.2

Also I tried make this settings at logstash.yml

xpack.monitoring.enabled: false
xpack.management.enabled: false

What I am doing wrong? Logstash and Elastic must have same version?
I replaced http with hzzp, because Forum says me, that I can’t put more than 2 link in my message

2

Elastic and Logstash version must be same

3

Hey @ThreatInter - this is a known thing. Take a look at the announcement about Logstash 7.13

4

@searchymcsearchface

to me it looks like that elastic have but a license check in version 7.13 of the OSS *beats / logstash effectively breaking interoperability with open search.

Looking at how opensearch came about I could see this as a move from elastic to have their OSS versions of *beats / logstash work with their OSS version of elastic - only.

what is your position on this? are you going to fork and maintain the *beats and logstash to in order to ensure support?

here is the pull that i believe is the “move” from elastic
https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1005
not being able to keep software updated as possible vunlerabilities arise etc, will in my view keep orgs from using open search, as they will have policies / requirements in place that requires them to keep software updated.

5

Hey @ssi0202. That’s known and people have been working on it for quite some time. Expect a release of a output plugin for Logstash that works with OpenSearch shortly (possibly today!). In the mean time, you can take a look at the source and build it yourself on github GitHub - opensearch-project/logstash-output-opensearch: A Logstash plugin that sends event data to a OpenSearch clusters and stores as an index.

Beats is more complicated - the way beats is put together doesn’t allow for a more graceful solution without drastic moves. However, this seems to only really be an issue when you are using Beats directly connected to OpenSearch, which is possible but not a very typical setup. If you’re using Beats → Logstash → OpenSearch, you should be fine after the output plugin.

6

hi @searchymcsearchface thanks for the quick reply. i will try it out when the plugin is done sounds like its just around the cornor.

so it would still be OSS *beats → OSS logstash (with the plugin) ->opensearch correct ?

im not sure i agree with the beats → opensearch is not that widely used, based on my experience, but you most likely have better data on that :wink:

7

Yup. Disclaimer: I’ve not personally used OSS beats → logstash → opensearch, but this is what I’ve been told should work. It’s on my list to write up (look for that soon).

If this is a greenfield project, I’d look at something like Fluentbit instead of Beats. My personal POV is that Fluentbit has a bright future.

WRT beats->directly to OpenSearch, I’d like to know more! Do you or someone you know use this setup?

8

Is there any news related to the problem from the beginning?

The logstash oss version was released a few days ago and we tested it within our testing lab, based on all opensearch components.

We also get the following error message, within logstash:

logstash               | [2021-08-16T11:37:26,979][ERROR][logstash.outputs.elasticsearch][main] Unable to get license information {:url=>"https://admin:xxxxxx@xx.xx.xx.xx:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '400' contacting Elasticsearch at URL 'https://xx.xx.xx.xx:9200/_license'"}
logstash               | [2021-08-16T11:37:26,979][ERROR][logstash.outputs.elasticsearch][main] Could not connect to a compatible version of Elasticsearch {:url=>"https://admin:xxxxxx@xx.xx.xx.xx:9200/"}

Which logstash version should be used with opensearch?

Also opened an bug report [BUG] Elasticsearch output fail - Could not connect to a compatible version of Elasticsearch · Issue #54 · opensearch-project/logstash-output-opensearch · GitHub

9

@thsul According to the documentation, 7.13.x

10

Thank you. Logstash is working now, the wrong plugin was used. Details can be found in the “issue” on github.

1 Like
11

Hi @searchymcsearchface
I tried to downgrade the Logstash version to 7.13.4 OSS build and still no luck connecting to Opendistro Elasticsearch (1.11.0).
Here’s my logstash config:

input {
    beats {
        port => 5000
        codec => "json_lines"
    }
}

output {
    elasticsearch {
        ilm_enabled => false
        user => "admin"
        password => "PASSWORD"
        ssl => true
        ssl_certificate_verification => false
        hosts => "https://127.0.0.1:9200"
        index => "test-%{+YYYY.MM.dd}"
        cacert => "/path/to/root-ca.pem"
       }
}

I tried this previously on Logstash OSS 7.14.0 and same error.

Error:

[2021-08-19T03:47:05,246][ERROR][logstash.outputs.elasticsearch][main] Could not connect to a compatible version of Elasticsearch {:url=>"https://admin:xxxxxx@127.0.0.1:9200/"}
[2021-08-19T03:47:10,253][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://admin:xxxxxx@127.0.0.1:9200/"}
[2021-08-19T03:47:10,265][ERROR][logstash.outputs.elasticsearch][main] Unable to get license information {:url=>"https://admin:xxxxxx@127.0.0.1:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '500' contacting Elasticsearch at URL 'https://127.0.0.1:9200/_license'"}
[2021-08-19T03:47:10,265][ERROR][logstash.outputs.elasticsearch][main] Could not connect to a compatible version of Elasticsearch {:url=>"https://admin:xxxxxx@127.0.0.1:9200/"}
12

Do you have the OpenSearch Output plugin? You’ll need that for any version of either OSS Elasticsearch or OpenSearch.

13

Nope. I did not have.
However, I installed the plugin and switched config from “elasticsearch” to “opensearch” in the output section and things started to work fine! I also removed the “ilm_enabled” field.
Thanks :slight_smile:

2 Likes