OpenSearch-Dashboards Received 500 "Internal Server Error" after we changed admin password and restart service

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 2.9.0
OpenSearch-Dashboards 2.9.0/Chrome Web Browser

Describe the issue:
Our opensearch cluster(On the physical machine) uses http, but when we change the password of the admin, we should use securityadmin.sh which must be run with https.
So we have to do like this:
1、select a node , set it to https and restart
2、use securityadmin.sh to change the password and set it back to http
3、restart the node
4、use script to synchronize password to the dashboards

After all this,when we reload the opensearch-dashboards page(we logined in before change the passord) in chrome,we assume that it should return http_code 401 or go to the login in page…However,it return 500 “Internal Server Error”. I think it is a very bad user experience and maybe a bug

BTW,when we use low version of Opensearch or Opendistro for Elasticsearch cluster,there is no need to set https when we want to change the password. This problem won’t happen On Elasticsearch(I think OpenSearch should be better compared with elasticsearch :crazy_face:

I know two ways to “solve” the problem:
1、refresh the cookie or set cookie invalid when reload the page
2、force the user to go to /app/login page
However,these two ways are quite unacceptable on our production environment…

Is there a better way to solve this promblem without change the source code of security plugin for opensearch 2.9.0,such as,allow http to change password or not return http_code 500 to dashboards…

Thanks

Configuration:
node.name: ${HOSTNAME}
node.roles: [“ingest”,“remote_cluster_client”,“data”,“cluster_manager”]
http.port: 9200
transport.tcp.port: 9300
bootstrap.memory_lock: true
path.data: /data01/opensearch/data
path.logs: /var/log/opensearch
plugins.security.disabled: false
plugins.security.ssl.transport.pemcert_filepath: ${HOSTNAME}.pem
plugins.security.ssl.transport.pemkey_filepath: ${HOSTNAME}-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.ssl.http.enabled: false
plugins.security.ssl.http.pemcert_filepath: ${HOSTNAME}.pem
plugins.security.ssl.http.pemkey_filepath: ${HOSTNAME}-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn: [“xxxxxxxx(secrecy)”]
plugins.security.nodes_dn: [“xxxxxxxxxxx(secrecy)”]
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”,“security_rest_api_access”]
plugins.security.system_indices.enabled: true
prometheus.cluster.settings: false
prometheus.indices: true
prometheus.nodes.filter: _all
network.bind_host: 0.0.0.0
network.publish_host: ${HOSTNAME}
cluster.name: OpenSearch
discovery.seed_hosts: [“bigdata-vm-xxxx”,“bigdata-vm-yyyy”,“bigdata-vm-zzzz”]
cluster.initial_master_nodes: [“bigdata-vm-xxxx”,“bigdata-vm-yyyy”,“bigdata-vm-zzzz”]

Relevant Logs or Screenshots:
{ “statusCode”: 500, “error”: “Internal Server Error”, “message”: “An internal server error occurred.” }

@pyyuhao The security plugin stores its configuration in the OpenSearch index. This index is protected and can be accessed only by super admin certificate defined in the admin_dn in opensearch.yml.
Since the SSL certificate is required to change the content of the security plugin index, the HTTPS endpoint must be enabled to establish SSL certificate authentication.
The securityadmin.sh only follows this logic and that’s why requires HTTPS to work.

@pablo Thanks for your reply

We understand and accept that it is forced to use https。
Can you help me figure out why 500 “Internal Server Error” occured after password is changed :

‘’’ when we reload the opensearch-dashboards page(we logined in before change the passord) in chrome,we assume that it should return http_code 401 or go to the login in page…However,it return 500 “Internal Server Error”. I think it is a very bad user experience and maybe a bug ? ‘’’

@pyyuhao In my experience, the HTTP 500 error most often occurs with another error. Have you noticed any errors in OpenSearch or OpenSearch Dashboards that precede the HTTP 500 error?

@pablo Thanks for your reply again
I also tried changing admin’s password through a https cluster, it ocurred also in dashboards。I followed the steps to change admin’s password in the page : Applying changes to configuration files - OpenSearch Documentation

And in the opensearch log, it only shows like this:
[2024-11-08T15:28:33,244][WARN ][o.o.s.a.BackendRegistry ] [bigdata-vm-xxxxxxx] Authentication finally failed for admin from 192.168.0.24:54858

in the opensearch-dashboards log,it shows like this:
{“type”:“error”,“@timestamp”:“2024-11-08T07:28:32Z”,“tags”:,“pid”:1606691,“level”:“error”,“error”:{“message”:“Internal Server Error”,“name”:“Error”,“stack”:“Error: Internal Server Error\n at HapiResponseAdapter.toInternalError (/usr/local/opensearch-dashboards-2.9.0_ccdp3.4.3_1.0.1/src/core/server/http/router/response_adapter.js:80:19)\n at Object.interceptAuth [as authenticate] (/usr/local/opensearch-dashboards-2.9.0_ccdp3.4.3_1.0.1/src/core/server/http/lifecycle/auth.js:151:34)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at exports.Manager.execute (/usr/local/opensearch-dashboards-2.9.0_ccdp3.4.3_1.0.1/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n at module.exports.internals.Auth._authenticate (/usr/local/opensearch-dashboards-2.9.0_ccdp3.4.3_1.0.1/node_modules/@hapi/hapi/lib/auth.js:273:30)\n at Request._lifecycle (/usr/local/opensearch-dashboards-2.9.0_ccdp3.4.3_1.0.1/node_modules/@hapi/hapi/lib/request.js:371:32)\n at Request._execute (/usr/local/opensearch-dashboards-2.9.0_ccdp3.4.3_1.0.1/node_modules/@hapi/hapi/lib/request.js:281:9)”},“url”:“http://{IP}:5601/app/dev_tools”,“message”:“Internal Server Error”}

After changing password,when we refresh the page,it shows like this:


I think opensearch service should not return 500 to dashboards…

We do think it is a bug when security plugin upgrade, you can test it in your environment, truly hope you can fix this

Thanks

@pyyuhao Could you share your opensearch_dashboards.yml file?
Also, share a response of these commands.

curl --insecure -u <username>:<password> https://<OpenSearch_FQDN_or_IP>:9200

curl --insecure -u <username>:<password> https://<OpenSearch_FQDN_or_IP>:9200/_plugins/_security/api/securityconfig?pretty

Sorry for replay so late. I have to deal other things these past days
As our service is http, so:

curl --insecure -u’admin:******’ -XGET http://localhost:9200

{
  "name" : "bigdata-vm-1728467696-wvc4t",
  "cluster_name" : "OpenSearch",
  "cluster_uuid" : "T5luglfzQnuu4X5I9Yn-6A",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.9.0",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2023-11-02T01:23:11.532157670Z",
    "build_snapshot" : false,
    "lucene_version" : "9.7.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

curl --insecure -u’admin:******’ -XGET http://localhost:9200/_plugins/_security/api/securityconfig?pretty

{
  "config" : {
    "dynamic" : {
      "filtered_alias_mode" : "warn",
      "disable_rest_auth" : false,
      "disable_intertransport_auth" : false,
      "respect_request_indices_options" : false,
      "kibana" : {
        "multitenancy_enabled" : true,
        "private_tenant_enabled" : true,
        "default_tenant" : "",
        "server_username" : "kibanaserver",
        "index" : ".kibana"
      },
      "http" : {
        "anonymous_auth_enabled" : false,
        "xff" : {
          "enabled" : false,
          "internalProxies" : "192\\.168\\.0\\.10|192\\.168\\.0\\.11",
          "remoteIpHeader" : "X-Forwarded-For"
        }
      },
      "authc" : {
        "jwt_auth_domain" : {
          "http_enabled" : false,
          "transport_enabled" : false,
          "order" : 0,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "jwt",
            "config" : {
              "signing_key" : "base64 encoded HMAC key or public RSA/ECDSA pem key",
              "jwt_header" : "Authorization",
              "jwt_clock_skew_tolerance_seconds" : 30
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "Authenticate via Json Web Token"
        },
        "ldap" : {
          "http_enabled" : false,
          "transport_enabled" : false,
          "order" : 5,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "basic",
            "config" : { }
          },
          "authentication_backend" : {
            "type" : "ldap",
            "config" : {
              "enable_ssl" : false,
              "enable_start_tls" : false,
              "enable_ssl_client_auth" : false,
              "verify_hostnames" : true,
              "hosts" : [
                "localhost:8389"
              ],
              "userbase" : "ou=people,dc=example,dc=com",
              "usersearch" : "(sAMAccountName={0})"
            }
          },
          "description" : "Authenticate via LDAP or Active Directory"
        },
        "basic_internal_auth_domain" : {
          "http_enabled" : true,
          "transport_enabled" : true,
          "order" : 4,
          "http_authenticator" : {
            "challenge" : true,
            "type" : "basic",
            "config" : { }
          },
          "authentication_backend" : {
            "type" : "intern",
            "config" : { }
          },
          "description" : "Authenticate via HTTP Basic against internal users database"
        },
        "proxy_auth_domain" : {
          "http_enabled" : false,
          "transport_enabled" : false,
          "order" : 3,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "proxy",
            "config" : {
              "user_header" : "x-proxy-user",
              "roles_header" : "x-proxy-roles"
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "Authenticate via proxy"
        },
        "clientcert_auth_domain" : {
          "http_enabled" : false,
          "transport_enabled" : false,
          "order" : 2,
          "http_authenticator" : {
            "challenge" : false,
            "type" : "clientcert",
            "config" : {
              "username_attribute" : "cn"
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          },
          "description" : "Authenticate via SSL client certificates"
        },
        "kerberos_auth_domain" : {
          "http_enabled" : false,
          "transport_enabled" : false,
          "order" : 6,
          "http_authenticator" : {
            "challenge" : true,
            "type" : "kerberos",
            "config" : {
              "krb_debug" : false,
              "strip_realm_from_principal" : true
            }
          },
          "authentication_backend" : {
            "type" : "noop",
            "config" : { }
          }
        }
      },
      "authz" : {
        "roles_from_another_ldap" : {
          "http_enabled" : false,
          "transport_enabled" : false,
          "authorization_backend" : {
            "type" : "ldap",
            "config" : { }
          },
          "description" : "Authorize via another Active Directory"
        },
        "roles_from_myldap" : {
          "http_enabled" : false,
          "transport_enabled" : false,
          "authorization_backend" : {
            "type" : "ldap",
            "config" : {
              "enable_ssl" : false,
              "enable_start_tls" : false,
              "enable_ssl_client_auth" : false,
              "verify_hostnames" : true,
              "hosts" : [
                "localhost:8389"
              ],
              "rolebase" : "ou=groups,dc=example,dc=com",
              "rolesearch" : "(member={0})",
              "userrolename" : "disabled",
              "rolename" : "cn",
              "resolve_nested_roles" : true,
              "userbase" : "ou=people,dc=example,dc=com",
              "usersearch" : "(uid={0})"
            }
          },
          "description" : "Authorize via LDAP or Active Directory"
        }
      },
      "auth_failure_listeners" : { },
      "do_not_fail_on_forbidden" : false,
      "multi_rolespan_enabled" : true,
      "hosts_resolver_mode" : "ip-only",
      "do_not_fail_on_forbidden_empty" : false
    }
  }
}

Also, I tested Opensearch 1.0.0、1.3.6、1.3.13,after changing the password of admin,they worked queite well,On Opensearch 1.3.16、1.3.19,it returned 401 http_code which I think is reluctantly ok for user’s experience(not as good as lower version).

Can you help me figure out the reason, mostly focused on why return http_code 500 after chaning admin’s password

@pyyuhao Do you use any reverse proxy in front of OpenSearch Dashboards?
Could you share opensearch_dashboards.yml content?

we have no reverse proxy, OpenSearch Dashboards has a fixed IP

server.host: "::"
server.port: 5601
path.data: ./data
logging.dest: /var/log/opensearch-dashboards/opensearch-dashboards.log
i18n.locale: zh-CN
opensearch.requestHeadersAllowlist: ["securitytenant", "Authorization"]
opensearch.hosts: ["http://bigdata-vm-1728467696-xxxx:9200","http://bigdata-vm-1728467696-xxxx:9200","http://bigdata-vm-1728467696-xxxx:9200"]

BTW, I think the problem is still about OpenSearch Security
Looking forward to your reply

Change this line to the following:

server.host: "0.0.0.0"

It will cause a problem with OpenSearch Dashboards.

Still,It returned 500


I’ve changed this config

Also,I tried almost ever version of OpenSearch+Dashboards from 1.0.0 to 2.18.0,I find out that 2.4.0 is the first version that the http_code 500 problem occurred,and after that version,it is the same problem…
from 1.0.0 to 2.3.0, it returned 401(which I think is reluctantly ok)

Hoping you can find the bug and fix it
Thanks and Looking forward to your reply

anything updated for the forum?
I met the same question for opensearch 2.11

@pkingqaq Did you check the updated password after running that script?
I understand that it replaces the value of opensearch.password option.

Why do you use admin user in opensearch_dashboards.yml?
Did you consider using kibanaserver instead? Admin user has too wide privileges for being a service account. You shouldn’t expose the admin password in any text files.

Did you test using different browsers and private mode? Try clearing the cookies.

@pablo I think more and more users will find this “bug” with the increasing use of OpenSearch and Dashboards :crazy_face:。As I tested,this promblem ocurred firstly on version 2.4.0。I know by erasing the cookie of the browser, it can fix this problem。Truly hopes that this issue can be fixed by the offical team.
Thanks