Security issue while dumping logs to opensearch dashboards

OpenSearch
1

I’m using logstash pipeline to collect and dump logs in Opensearch dashboard. The pipelines.yml file reads like this :

input {
file {
path => “/var/log/opensearch/*”
}
}
filter {
json {
source => “message”
}
}

output {
opensearch {
hosts => [“localhost:9200”]
index => “logstash-logs-%{+YYYY.MM.dd}”
}
}

The pipeline works fine when I remove ‘s’ out of https, and make plugins.security.disabled=true in opensearch.yml.

It threw following error when I tried with http by commenting the “plugins.security.disabled=true” portion in opensearch.yml.

[2023-11-09T11:56:09,189][WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>“OpenSearch Unreachable: [http://localhost:9200/][Manticore::ClientProtocolException] localhost:9200 failed to respond”

If I need a secure connection, what should I do?

2

@NandanIK When you enable security plugin then you must connect with HTTPS protocol to OpenSearch node.

How did you deploy the cluster?

Is your Logstash running at the same host/vm as OpenSearch nodes?

If your OpenSearch cluster is using self signed certificates then you need to turn off certificate verification or present RootCA of the OpenSearch node in the OpenSearch output’of Logstash config.

3

Hello @pablo , Thanks for response, yes…!, our 1] Logstash and OpenSearch Running on same Virtual machine, 2] Can you please guide how to turn off self signed certificates,

4 
######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
#plugins.security.ssl.transport.pemcert_filepath: esnode.pem
#plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
#plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
#plugins.security.ssl.transport.enforce_hostname_verification: false
#plugins.security.ssl.http.enabled: true
#plugins.security.ssl.http.pemcert_filepath: esnode.pem
#plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
#plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
#plugins.security.allow_unsafe_democertificates: true
#plugins.security.allow_default_init_securityindex: true
#plugins.security.authcz.admin_dn:
#  - CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.disabled: false
#plugins.security.audit.type: internal_opensearch
#plugins.security.enable_snapshot_restore_privilege: true
#plugins.security.check_snapshot_restore_write_privileges: true
#plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-config", ".plugins-ml-connector", ".plugins-ml-model-group", ".plugins-ml-model", ".plugins-ml-task", ".plugins-ml-conversation-meta", ".plugins-ml-conversation-interactions", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".ql-datasources", ".opendistro-asynchronous-search-response*", ".replication-metadata-store", ".opensearch-knn-models", ".geospatial-ip2geo-data*"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/node1.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/node1.pem
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - 'CN=A,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
plugins.security.nodes_dn:
  - 'CN=node1.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]



``opensearch.yml file look like this
5

image
image805×525 13.6 KB

6

@Afzal Try the below config with security plugin enabled in OpenSearch.

opensearch {
        hosts => ["https://localhost:9200"]
        # SSL enabled
        ssl => true
        ssl_certificate_verification => false
        # The Logstash Username and Password created earlier
        user => ""
        password => ""
        # The name of the index
        index => "<index_name>"
    }
7

Hello @pablo , Thanks for the Support, I will try it for sure

8

Hello @pablo , I need one more help, whenever I tried for pushing data to OpenSearch, I am getting this line for logstash run - "Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>} " Not able to push the logs into opensearch
I request please help me on this,

9

@Afzal What user did you use?