Hi!
I’m trying to configure logstash to output to opensearch. I have few questions.
- how to install plugin?
I tried to use gem
logstash-1:/opt/logstash/config/conf.d # gem 'logstash-output-opensearch', '~> 1.2'
ERROR: While executing gem ... (Gem::CommandLineError)
Unknown command logstash-output-opensearch,
/usr/lib64/ruby/2.5.0/rubygems/command_manager.rb:184:in `find_command'
/usr/lib64/ruby/2.5.0/rubygems/command_manager.rb:172:in `process_args'
/usr/lib64/ruby/2.5.0/rubygems/command_manager.rb:143:in `run'
/usr/lib64/ruby/2.5.0/rubygems/gem_runner.rb:59:in `run'
/usr/bin/gem:21:in `<main>'
I don’t know gem system and how it works.
- In machines I will not have access to the internet, how to install without it?
- I have configured TLS and SSL on opensearch cluster, how to configure with it?
Security configuration opensearch:
plugins.security.ssl.transport.pemcert_filepath: sa5uts-opm-1.pem
plugins.security.ssl.transport.pemkey_filepath: sa5uts-opm-1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: sa5uts-opm-1.pem
plugins.security.ssl.http.pemkey_filepath: sa5uts-opm-1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
- 'CN=ADMIN,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
plugins.security.nodes_dn:
- 'CN=sa5uts-op*,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
node.max_local_storage_nodes: 3
Just use login/pass with ssl_certificate_verification => false ?
Thank you!
Hi @mamol27 -
You can download a version built by us that contains the opensearch output plugin by default by following the instructions here: Logstash - OpenSearch documentation
It should address some of your concerns - you won’t have to install any gems, and you can always keep the tarball some place on your LAN where your machines can get at it without having to have access to the internet at large.
What you do with your security configuration depends on a few things. You provided your configuration, but are there any errors you get when you try to run it? Which part isn’t working for you?
If your certificates are self signed, you will likely have to use the configuration directive you mentioned - ssl_certificate_verification: false as self signed certificates will never be automatically trusted by any root certificate authority.
I hope that helps. Let us know what errors you’re getting with your current security configuration and let’s see if we can conquer those.
Nate
There is my setting Logstash output into elasticsearch, which configured the same way.
elasticsearch {
hosts => ["eld-11:9200", "eld-12:9200", "eld-13:9200"]
ssl => true
cacert => "/etc/certificates/salt-ca.crt"
user => logstash_internal
password => "superpass"
manage_template => false
index => "kafka-%{+YYYY.MM.dd}"
}
As you see, I use ca certificate and login/pass.
How to configure logstash to output to opensearch with enabled SSL? I have root ca and certificates for logstash.
Sorry @mamol27 - I’m still not quite sure what’s going on.
If you could please provide some error messages we could use to troubleshoot which part is going wrong, that would certainly help. Is OpenSearch and OpenSearch Dashboards up and running already with working certificates? Do you have certificate errors when visiting your dashboards page?
I can’t quite understand which part
Can you get logstash on its own to run with the most basic configuration possible? I.e.
bin/logstash -e "input { stdin { } } output { stdout { } }"
This tells logstash to use STDIN as the input (your keyboard), and output to STDOUT (the screen). Try running logstash like this and then typing some text into the screen to see if it generates a JSON message.
Here’s the example provided from Logstash - OpenSearch documentation :
input { stdin { } } output {
opensearch {
hosts => ["https://opensearch:9200"]
index => "opensearch-logstash-docker-%{+YYYY.MM.dd}"
user => "admin"
password => "admin"
ssl => true
ssl_certificate_verification => false
}
}
Let’s see if we can get things narrowed down a bit more. Please do visit
Logstash - OpenSearch documentation and see if any of the example configs there help you.
Nate
@nateynate
output {
opensearch {
hosts => ["https://172.29.39.181:9200","https://172.29.39.179:9200"]
user => "admin"
password => "admin"
index => "mr-filebeat-%{+YYYY.MM.dd}"
ssl => true
ssl_certificate_verification => false
}
}
[2022-04-11T11:13:39,802][WARN ][logstash.outputs.opensearch][main] ** WARNING ** Detected UNSAFE options in opensearch output configuration!
** WARNING ** You have enabled encryption but DISABLED certificate verification.
** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
if i change :ssl_certificate_verification to true i will see next:
output {
opensearch {
hosts => ["https://172.29.39.181:9200","https://172.29.39.179:9200"]
user => "admin"
password => "admin"
index => "mr-filebeat-%{+YYYY.MM.dd}"
ssl => true
ssl_certificate_verification => true
}
}
[2022-04-11T13:16:42,163][WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"https://admin:xxxxxx@172.29.39.179:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [https://admin:xxxxxx@172.29.39.179:9200/][Manticore::ClientProtocolException] sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
[2022-04-11T13:16:47,200][WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"https://admin:xxxxxx@172.29.39.181:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [https://admin:xxxxxx@172.29.39.181:9200/][Manticore::ClientProtocolException] sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
I need to use connection in secure mod. As I show setting for elasticsearch I use CA certificate. I didn’t find any security configuration for output opensearch in your documentation.
How to configure security connection between logstash and opensearch.
Opensearch and opensearch dashboards works with SSL and TLS configuration, as I show.
Thanks @mamol27 - I have a hunch about something here.
You’re referring to your OpenSearch hosts by IP address instead of hostname, which will fail certificate validation every time because they cover hostnames, not IP addresses.
Try referring to your nodes via a hostname that is covered by your certificate and let’s see how if the error goes away or not.
Nate
@nateynate I changed ip to hostnames
output {
opensearch {
hosts => ["https://sa5uts-opd-1:9200","https://sa5uts-opm-1:9200"]
user => "admin"
password => "admin"
index => "mr-filebeat-%{+YYYY.MM.dd}"
ssl => true
ssl_certificate_verification => true
}
}
[2022-04-12T10:32:43,323][WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"https://admin:xxxxxx@sa5uts-opd-1:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [https://admin:xxxxxx@sa5uts-opd-1:9200/][Manticore::ClientProtocolException] sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
[2022-04-12T10:32:43,355][WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"https://admin:xxxxxx@sa5uts-opm-1:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [https://admin:xxxxxx@sa5uts-opm-1:9200/][Manticore::ClientProtocolException] sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
I have a premonition that I need to use certificate like in elasticsearch output, but I can’t find settings for certificate path.
A “host unreachable” error sounds more like a network connection issue than an SSL issue, but I think there’s a few more pertinent questions here.
The hostname change may not be all the way complete yet. SSL certificates cover a full domain name, or in the case of a wildcard certificate, multiple bottom level domains under the same top level domain. I don’t think a certificate can cover just ‘sa5uts-opd-1’ - what hostname / wildcard does your certificate cover? Make sure that the hostname you’re using to access it matches.
About which certificate are y talking? I didn’t set up any certificate into opensearch output plugin. I have certificates on my cluster, which setting I showed to you in my first block, and it works fine.
I have generated certificates as showed here Generate certificates - OpenSearch documentation
This error causes after I’m set up ssl_certificate_verification => true with turned off it works, but without SSL, and I need it with SSL.