Integrate Logstash or fluentd with Open Distro for Elasticsearch

Hi,

I am trying to use logstash and fluentd in two different Instances to test logs forwarding. However I am get in to some issues which are related to SSL certificates. I could see the logs are receiving to the OpenDistro node in a tcpdump, but nothing has been inserted to the Elasticsearch.

should I need to use https instead of http in all over the configurations? Can you please share a little guide to fix this configuration issue?

Thank you
Luke

1 Like

Is you issue resolved, I have same issue with you. If you fixed your issue, please post it in this topic. thanks.

Hi,

Yes, I managed to resolve the issue. Try the following config in your td-agent.conf

<match *.**>
@type elasticsearch
hosts https://admin:admin@localhost:9200
scheme https
ssl_verify false
ssl_version TLSv1_2
logstash_format true
logstash_prefix fluentd
enable_ilm true
index_date_pattern “now/m{yyyy.mm}”
flush_interval 10s

Cheers,
Luke.

Hi Luke,

Can you share your logstash config and Filebeat config like below?

Logstash config:

input {

file {

path => “/var/log/messages”

start_position => “beginning”

}

}

filter {

}

output {

elasticsearch {

hosts => [“https://192.168.4.231:9200”]

index => “messages-%{+YYYY-MM-dd}”

user => admin

password => admin

ssl => true

ssl_certificate_verification => false

ilm_enabled => false

cacert => “/etc/logstash/root-ca.pem”

}

Filebeat Config:

filebeat.config.modules:

enabled: true

path: ${path.config}/modules.d/*.yml

setup.ilm.overwrite: true

setup.ilm.enabled: false

output.elasticsearch:

ssl.verification_mode: none

hosts: [“localhost”]

username: “admin”

password: “admin”

setup.kibana:

host: “localhost”

logging.level: debug

logging.to_files: true

logging.files:

path: /var/log/filebeat

name: filebeat

keepfiles: 7

permissions: 0644

Best Regards,

Stone Liu

@luke_devon
Sorry to spam some older posts - but wanted to mention for fluentd - please look to the newly released fluent-plugin-opensearch
link: GitHub - fluent/fluent-plugin-opensearch: OpenSearch Plugin for Fluentd