Opensearch Dashboards Security Plugin causes the Dashboards to stop working (ECONNREFUSED)

Security
1

Hi,

I’m running Opensearch v2.2.0 with self signed certs for the OS security plugin.

The configured ldap authc & authz in config.yml works fine and I’m able to run https REST API based querys, but since setting the “plugins.security.disabled: false” in opensearch.yml & installation of the OS dashboards security plugin - all dashboard logins fails with an
“[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200” error in /var/log/opensearch/opensearch-dashboard log.

If I set the “plugins.security.disabled: true” setting back and remove the /usr/opensearch-dashboards/plugins/securityDashboards plugin the OS Dashboards starts working again, but then REST API querys cannot run through https since it requires the security plugin

Do anyone know how to resolve this issue?

opensearch_dashboards.yml config. below,

server.host: “127.0.0.1”
opensearch.hosts: [http:/127.0.0.1:9200]
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: kibanaserver
opensearch.requestHeadersAllowlist: [authorization, securitytenant]
opensearch_security.multitenancy.enabled: false
opensearch_security.multitenancy.tenants.enable_global: true
opensearch_security.multitenancy.tenants.enable_private: true
opensearch_security.multitenancy.tenants.preferred: [“Private”, “Global”]
opensearch_security.multitenancy.enable_filter: false
opensearch_security.readonly_mode.roles: [kibana_read_only]
logging.dest: /var/log/opensearch/opensearch-dashboard.log
logging.quiet: false
server.ssl.enabled: false

opensearch.yml below,

plugins.security.ssl.transport.pemcert_filepath: ./cert/node1.pem
plugins.security.ssl.transport.pemkey_filepath: ./cert/node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: ./cert/root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: ./cert/node1.pem
plugins.security.ssl.http.pemkey_filepath: ./cert/node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: ./cert/root-ca.pem
plugins.security.ssl.http.clientauth_mode: OPTIONAL
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
‘CN=XX,OU=XX,O=XX,L=XX,ST=XX,C=SE’
plugins.security.nodes_dn:
‘CN=XX,OU=XX,O=XX,L=XX,ST=XX,C=SE’
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [“.plugins-ml-model”, “.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opensearch-notifications-”, “.opensearch-notebooks”, “.opensearch-observability”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”]
node.max_local_storage_nodes: 3
plugins.security.disabled: false

parts of config.yml below,

  basic_internal_auth_domain:
    description: "Authenticate via HTTP Basic against internal users database"
    http_enabled: true
    transport_enabled: true
    order: 0
    http_authenticator:
      type: basic
      challenge: true
    authentication_backend:
      type: intern

  ldap:
    description: "Authenticate via LDAP or Active Directory"
    http_enabled: true
    transport_enabled: true
    order: 4
    http_authenticator:
      type: basic
      challenge: true

authz:
  roles_from_myldap:
    description: "Authorize via LDAP or Active Directory"
    http_enabled: true
    transport_enabled: true
    authorization_backend:
      type: ldap
      config:
2

@denkar You have the following set in opensearch_dashboards.yml file.

opensearch.hosts: [http:/127.0.0.1:9200]

When security plugin is enabled and plugins.security.ssl.http.enabled set to true the only available communication is over HTTPS.
Either use https:/127.0.0.1:9200 in opensearch_dashboards.yml or set false in plugins.security.ssl.http.enabled.

Please be aware that plugins.security.ssl.http.enabled has to be set to true if you want to use securityadmin.sh script.

3

@pablo
Thank you for helping out, but now after I made the change from http to https I get a Proxy error in the browser.

Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
Reason: Error reading from remote server

I also added my self signed certs to the opensearch_dashboards.yml file and enabled ssl, se below

server.host: “127.0.0.1”

opensearch.hosts: [https:/xxx.xxx.xx:9200]
opensearch.ssl.verificationMode: certificate
opensearch.username: kibanaserver
opensearch.password: kibanaserver
opensearch.requestHeadersAllowlist: [authorization, securitytenant]
opensearch_security.multitenancy.enabled: false
opensearch_security.multitenancy.tenants.enable_global: true
opensearch_security.multitenancy.tenants.enable_private: true
opensearch_security.multitenancy.tenants.preferred: [“Private”, “Global”]
opensearch_security.multitenancy.enable_filter: false
opensearch_security.readonly_mode.roles: [kibana_read_only]
opensearch.ssl.certificateAuthorities:
[/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem]
server.ssl.enabled: true
server.ssl.certificate: /usr/opensearch/config/cert/node1.pem
server.ssl.key: /usr/opensearch/config/cert/node1-key.pem

logging.dest: /var/log/opensearch/opensearch-dashboard.log
logging.quiet: false

(Tried with setting opensearch.hosts: to 127.0.0.1 and to the cn specified in opensearch.yml under plugins.security.nodes_dn but both fails.)

log error messages:
{“type”:“log”,“@timestamp”:“2022-10-18T09:54:40Z”,“tags”:[“info”,“plugins-system”],“pid”:1118,“message”:“Setting up [46] plugins: [alertingDashboards,mapsLegacy,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,share,opensearchUiShared,legacyExport,expressions,data,home,apmOss,console,management,indexPatternManagement,advancedSettings,savedObjects,reportsDashboards,securityDashboards,anomalyDetectionDashboards,indexManagementDashboards,queryWorkbenchDashboards,notificationsDashboards,embeddable,dashboard,visualizations,visTypeVega,visTypeMarkdown,visTypeTimeline,timeline,visTypeTable,tileMap,regionMap,customImportMapDashboards,inputControlVis,ganttChartDashboards,visualize,bfetch,charts,visTypeVislib,visTypeMetric,visTypeTimeseries,visTypeTagcloud,observabilityDashboards,discover,savedObjectsManagement]”}
{“type”:“log”,“@timestamp”:“2022-10-18T09:54:41Z”,“tags”:[“info”,“savedobjects-service”],“pid”:1118,“message”:“Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations…”}
{“type”:“log”,“@timestamp”:“2022-10-18T09:54:41Z”,“tags”:[“error”,“opensearch”,“data”],“pid”:1118,“message”:“[ConnectionError]: connect ECONNREFUSED 10.126.2.108:9200”}
{“type”:“log”,“@timestamp”:“2022-10-18T09:54:41Z”,“tags”:[“error”,“savedobjects-service”],“pid”:1118,“message”:“Unable to retrieve version information from OpenSearch nodes.”}
{“type”:“log”,“@timestamp”:“2022-10-18T09:54:44Z”,“tags”:[“error”,“opensearch”,“data”],“pid”:1118,“message”:“[ConnectionError]: connect ECONNREFUSED 10.126.2.108:9200”}

But the https REST API works…

A small example from an https REST API query below,
curl -v -XGET “https://xxx.xxx.xxx:9200/_plugins/_security/api/rolesmapping” -u xx:xx -ks | jq

query result:

“all_access”: {
“hosts”: [
“"
],
“users”: [
"”
],
“reserved”: false,
“hidden”: false,
“backend_roles”: [
“ldap_test”
],
“and_backend_roles”:
},

4

@denkar

server.host: “127.0.0.1”

With that setting OpenSearch Dashboards will be available only from localhost?

Do you run Opensearch and OpenSearch Dashboards on the same machine?
Could you describe your OpenSearch infrastructure?

5

Please check this link. It describes how to use opensearch.ssl.verificationMode and opensearch.ssl.certificateAuthorizites.

6

@pablo

Opensearch and Opensearch dashboards are installed on the same host.
(A linux standalone VM with Apache)

7

@denkar I understand that you access the OpenSearch Dashboards directly from that Linux VM, am I correct?

8

@pablo

I access the Opensearch env. via my web browser from another client machine in the same network.

Client <–> https <–> VM Apache ProxyPass / http://127.0.0.1:5601/ <–> Opensearch Dashboards

Dashboard access works fine as long as the security plugin is disabled and the securityDashboards are removed from the /usr/opensearch-dashboards/plugins directory

9

@pablo

The only thing that does not work is the https access to the OS Dashboards when REST API https (with self signed certs) is enabled, otherwise it works.

Do you see any changes that might need to be made in the opensearch_dashboards.yml file or in the config.yml file in order to make this work?

10

@denkar I wasn’t aware of the proxy.
This section:

server.ssl.enabled: true
server.ssl.certificate: /usr/opensearch/config/cert/node1.pem
server.ssl.key: /usr/opensearch/config/cert/node1-key.pem

as per documentation is used for the browser - OSD connection.

In your case proxy acts as a client for OSD. To keep the current configuration of your proxy you should remove/comment out the above lines.

Regarding the connection between OSD and OS, can you try to use the below?

opensearch.ssl.verificationMode: none

If you’d like to use certificate instead of none, then the certificate defined in opensearch.ssl.certificateAuthorities must be exactly the same as the one defined in plugins.security.ssl.http.pemtrustedcas_filepath in opensearch.yml.

11

@pablo

I have disabled the Apache proxy service just too keep it out of the equation.

I also changed the OSD server.host from 127.0.0.1 to 10.126.2.108 to make OSD accessible over the network.
The ssl cert paths now also match the settings I used to get the REST API https working.

My new opensearch_dashboards.yml below,

server.host: “10.126.2.108”
opensearch.hosts: [https://127.0.0.1:9200]
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: kibanaserver
opensearch.requestHeadersAllowlist: [authorization, securitytenant]

logging.dest: /var/log/opensearch/opensearch-dashboard.log
logging.quiet: false

opensearch_security.multitenancy.enabled: false
opensearch_security.multitenancy.tenants.enable_global: true
opensearch_security.multitenancy.tenants.enable_private: true
opensearch_security.multitenancy.tenants.preferred: [“Private”, “Global”]
opensearch_security.multitenancy.enable_filter: false
opensearch_security.readonly_mode.roles: [kibana_read_only]
opensearch.ssl.certificateAuthorities:
[/usr/opensearch/config/cert/root-ca.pem]

server.ssl.enabled: true
server.ssl.certificate: /usr/opensearch/config/cert/node1.pem
server.ssl.key: /usr/opensearch/config/cert/node1-key.pem

Restarted the server but I still get the connection errors…

{“type”:“log”
,“@timestamp”:“2022-10-18T19:05:01Z”,“tags”:[“error”,“opensearch”,“data”],“pid”:1119,“message”:“[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200”}

12

@pablo

Thank you for all the help, finally managed to got it working.

For anyone who is interested or are having the same type of problem, my version of opensearch_dashboard.yml is below,

server.host: “127.0.0.1”
opensearch.hosts: [https://localhost:9200]
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: kibanaserver
opensearch.requestHeadersAllowlist: [ authorization,securitytenant ]

logging.dest: /var/log/opensearch/opensearch-dashboard.log
logging.quiet: false

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.enable_global: true
opensearch_security.multitenancy.tenants.enable_private: true
opensearch_security.multitenancy.tenants.preferred: [“Private”, “Global”]
opensearch_security.multitenancy.enable_filter: false
opensearch_security.readonly_mode.roles: [kibana_read_only]
opensearch.ssl.certificateAuthorities: [“/usr/opensearch/config/cert/root-ca.pem”]

#server.ssl.enabled: true
#server.ssl.certificate: /usr/opensearch/config/cert/localhost.pem
#server.ssl.key: /usr/opensearch/config/cert/localhost-key.pem
#opensearch_security.cookie.secure: true

13

@pablo

Now when OSD and REST API is working via SSL/TLS I noticed that the logstash output stream doesnt work although I added my self signed root cert to the output config. (Logstash is Installed on a separate linux VM)
root-ca.pem file permissions are set the same as on the OS machine, and there are no firewall restrictions between the machines.

Any idea what I may be missing?

Logstash output config. is below,

output {
opensearch {
hosts => [“https://xx.xx.xx:9200”]
ssl => true
ssl_certificate_verification => false
cacert => ‘/usr/logstash-8081/cert/root-ca.pem’
user => xxx
password => xxx
index => “%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}”
}
}

14

@denkar You can remove cacert from the config when you have ssl_certificate_verification set to false.

output {
   opensearch {
      hosts => [“https://xx.xx.xx:9200”]
      ssl => true
      ssl_certificate_verification => false
      user => xxx
      password => xxx
      index => “%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}”
  }
}

The user must have either logstash backend role assigned

image
image583×765 44.9 KB

or be an admin.

However, I would avoid using an admin user in this scenario.

1 Like
15

@pablo

I created a new internal user and assigned it the logstash backend role,
and modified the logstash output section to:

output {
opensearch {
hosts => [“https://xx.xx.xx:9200”]
ssl => true
ssl_certificate_verification => false
user => “XX”
password => “XX”
index => “%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}”
}
}

Then I started up the logstash service, and noticed that no errors are registered in the logstash log file, which looks fine - but no events arrive in Opensearch and no new index gets created.

A short excerpt from the logstash log below,

[2022-10-19T19:32:19,617][INFO ][logstash.javapipeline ][beats] Pipeline Java execution initialization time {“seconds”=>0.83}
[2022-10-19T19:32:19,636][INFO ][logstash.inputs.beats ][beats] Starting input listener {:address=>“0.0.0.0:8081”}
[2022-10-19T19:32:19,811][INFO ][logstash.javapipeline ][beats] Pipeline started {“pipeline.id”=>“beats”}
[2022-10-19T19:32:19,870][INFO ][org.logstash.beats.Server][beats][687ac856222a272055e527225e22104051e5d7268d41b584259f23eb41def7aa] Starting server on port: 8081
[2022-10-19T19:32:20,131][WARN ][logstash.outputs.opensearch][filebeat-message-pl] Restored connection to OpenSearch instance {:url=>“https://xxxxx:xxxxxx@xx.xx.xx:9200/”}
[2022-10-19T19:32:20,205][INFO ][logstash.outputs.opensearch][filebeat-message-pl] Cluster version determined (2.2.0) {:version=>2}
[2022-10-19T19:32:20,262][INFO ][logstash.outputs.opensearch][filebeat-message-pl] Using a default mapping template {:version=>2, :ecs_compatibility=>:disabled}
[2022-10-19T19:32:20,549][INFO ][logstash.javapipeline ][filebeat-message-pl] Starting pipeline {:pipeline_id=>“filebeat-message-pl”, “pipeline.workers”=>4, “pipeline.batch.size”=>125, “pipeline.batch.delay”=>50, “pipeline.max_inflight”=>500, “pipeline.sources”=>[“/usr/logstash-8081/config/conf.d/02-filebeat-message-pipeline.conf”], :thread=>“#<Thread:0x77170a0d run>”}
[2022-10-19T19:32:21,167][INFO ][logstash.javapipeline ][filebeat-message-pl] Pipeline Java execution initialization time {“seconds”=>0.62}
[2022-10-19T19:32:21,185][INFO ][logstash.javapipeline ][filebeat-message-pl] Pipeline started {“pipeline.id”=>“filebeat-message-pl”}
[2022-10-19T19:32:21,205][INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:beats, :“filebeat-message-pl”], :non_running_pipelines=>}

Do I need to specify my self signed root cert and client cert + key in the output section to get this working with the OS security plugin ?

16

@denkar With ssl_certificate_verification => false in the output you don’t need to specify any certificates.
Maybe either your filtering in logstash doesn’t work or filebeat doesn’t send any data to logstash.
Try testing with the admin user or assign the admin backend role to the existing logstash user.

Since logstash can connect to OpenSearch, this is no longer a security plugin issue.

[2022-10-19T19:32:20,131][WARN ][logstash.outputs.opensearch][filebeat-message-pl] Restored connection to OpenSearch instance {:url=>“https://xxxxx:xxxxxx@xx.xx.xx:9200/”}

Try following OpenSearch documentation for further Logstash troubleshooting.

17

@pablo

You were right, my logstash vm was not sending any data
Now everything works - thank you for all the help in this troubleshooting

18

@pablo

Is it possible to hide the Security plugin menu in OSD from standard readonly users?
(I mean the whole “Security” menu option under Opensearch Plugins → Security)

19

@denkar I’ve just created a user with a kibana_user role only and the Security tab wasn’t there.
image

20

@pablo

ok, perfect.
Now it looks really good for ro users