Using Logstash in Opendistro

Open Source Elasticsearch and Kibana
1

Hi team,

How to use logstash in opendistro.Can we use ELK logstash 7.8.0(Logstash 7.8.0 | Elastic) with Opendistro 1.9.0 ?

I want to extract/download index records in opendsitro elasticsearch into csv file using logstash in opendistro and I am getting below errors.

logstash config file:
input {
elasticsearch {
hosts => “https://127.0.0.1:9200”
index => “mvp1sandbox1”
query => ’
{
“query”: {
“match_all”: {}
}
}
’
}
}

output {
csv {

fields => [“amountTransactionPerCard_0”,“amountTransactionPerCard_2”]
path => “D:/Pooja_kibana/lostash data trigger/opndstrlog.csv”
}
}

command: D:\project\opendistrojdbc\logstash-7.8.0\bin>logstash -f “D:\Project\opendistrojdbc\logstash-7.8.0\config\logstash-sample.conf”

Errors:

[2020-09-11T00:54:06,314][ERROR][logstash.javapipeline ][main][78a6e46a2c13fb0ccd23ce773eec1433e1fc2658f10c18c6a5310b8d2245862d] A plugin had an unrecoverable error. Will restart this plugin.

Error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Exception: Manticore::ClientProtocolException

Please suggest…

2

Hello @sarvendras,

Yes, Logstash works with OD. But you probably should consider to move to the Logstash OSS edition.

About your error - looks like the issue with the communication. Accordingly to the documentation there should specified that communication is done over ssl - Elasticsearch input plugin.

3

@stmx38
Thanks for the reply… I I have used the logstash OSS edition(logstash-7.8.0) for opendistro (odfe-1.9.0) too but still getting same error.

Error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Exception: Manticore::ClientProtocolException

I am using below config file.Please suggest is it correct.I found no docuemntation of logstash on opendistro site.

input {
elasticsearch {
hosts => “https://127.0.0.1:9200”
index => “mvp1sandbox1”
user => “admin”
password => “admin”
}
}

output {
csv {
fields => [“amountTransactionPerCard_0”,“amountTransactionPerCard_2”]
path => “D:/Pooja_kibana/lostash_data_trigger/opndstrlog.csv”
}
}

4

@sarvendras, lets start from documentation - Elasticsearch input plugin | Logstash Reference [8.11] | Elastic

hosts

List of one or more Elasticsearch hosts to use for querying. Each host can be either IP, HOST, IP:port, or HOST:port. The port defaults to 9200.

You use hosts => "https://127.0.0.1:9200", probably should be hosts => "127.0.0.1:9200"?

5

@stmx38 Thanks a lot :slight_smile: …I will try as you suggested…

Just one more question is there any difference between in api(specially bulk update and update by query) of Elasticsearch and opendistro .Does both have exactly same sets of API.
Actually we need to replace ELK with opendistro so that’s why we need to know this.

Thanks
Sarvendra

6

OD uses Elasticsearch OSS - it should be the same Elasticsearch but just with free components and with same API. OD just add some additional functionality using plugins

7

Ok…Thank you…what is the difference between Elasticsearch and Elasticsearch OSS, Log stash and log stash OSS.