Dashboard container restart loop

OpenSearch
, ,
1

Versions (relevant - OpenSearch 2.18.0/Dashboard 2.18.0 /Server OS Rocky9.4):

Describe the issue:
We’ve recently upgraded to OpenSearch 2.18.0 and deployed the new docker images. When I run a docker ps query I get 2 pods returned. OpenSearch is find and has been running for 7 days (when it was created) but the OpenSearch-dashboard is “Restarting” and never progresses, rather it simply times out (I assume) after 60 seconds and starts again.

If I check the docker logs I see:
docker logs d9e7bff0ded1

{"type":"log","@timestamp":"2024-12-09T11:56:12Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"applicationConfig\" is disabled."}
{"type":"log","@timestamp":"2024-12-09T11:56:12Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"cspHandler\" is disabled."}
{"type":"log","@timestamp":"2024-12-09T11:56:12Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"dataSource\" is disabled."}
{"type":"log","@timestamp":"2024-12-09T11:56:12Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2024-12-09T11:56:12Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"workspace\" is disabled."}
{"type":"log","@timestamp":"2024-12-09T11:56:13Z","tags":["info","dynamic-config-service"],"pid":1,"message":"registering middleware to inject context to AsyncLocalStorage"}
[agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
[Error [ObjectUnsubscribedError]: object unsubscribed]
[agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead

 FATAL  TypeError: Cannot read properties of undefined (reading 'append')

I’m not sure where to go next with this one. I know I need to deal with the FATAL error, but not sure why I’m getting it…

Configuration:

Relevant Logs or Screenshots:

2

The configuration is not displaying for me. Can you post the redacted contents of opensearch_dashboards.yml?

3

Thanks for getting back to me!
As I understand it (no expert here!) the opensearch_dashboards.yml exists within the docker container, I’m unable to get the container started to review it, unless I’m barking up the wrong tree…

4

I’ve managed to find the openseach_dashboards.yml file, redacted contents are:

[<<>>@<<>> config]# cat opensearch-dashboards.yml

server.name: osd
server.host: '0.0.0.0'

opensearch.username: kibanaserver
opensearch.password: <<>>

opensearch.hosts:
    [https://opensearch-node01:9200]

# Encrypt traffic between the browser and Opensearch dashboards
server.ssl.enabled: true
server.ssl.certificate: /usr/share/opensearch-dashboards/config/certificates/opensearch-dashboards.cer
server.ssl.key: /usr/share/opensearch-dashboards/config/certificates/opensearch-dashboards.key

# Encrypt traffic between OpenSearch dashboards and Opensearch
opensearch.ssl.certificateAuthorities:
    [/usr/share/opensearch-dashboards/config/certificates/<<>>.cer]
opensearch.ssl.verificationMode: full

opensearch.requestHeadersAllowlist: ['securitytenant', 'Authorization']

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.enable_global: true
opensearch_security.multitenancy.tenants.enable_private: false
opensearch_security.multitenancy.tenants.preferred: ['Global']
opensearch_security.multitenancy.enable_filter: false

opensearch_security.auth.multiple_auth_enabled: true
# opensearch_security.auth.type: ["basicauth","openid"]
opensearch_security.auth.type: ["openid"]
opensearch_security.allow_client_certificates: true
opensearch_security.openid.connect_url: https://<<>>/.well-known/openid-configuration
opensearch_security.openid.base_redirect_url: https://<<>>
opensearch_security.openid.client_id: https://<<>>
opensearch_security.openid.client_secret: '<<>>'
opensearch_security.openid.root_ca: /usr/share/opensearch-dashboards/config/certificates/<<>>.cer
opensearch_security.openid.verify_hostnames: 'false'

opensearch_security.ui.basicauth.login.brandimage: /usr/share/opensearch-dashboards/config/dick.png
opensearch_security.ui.basicauth.login.showbrandimage: true

opensearch_security.ui.openid.login.buttonname: "<<>>"
opensearch_security.ui.openid.login.brandimage: /usr/share/opensearch-dashboards/config/dick.png
opensearch_security.ui.openid.login.showbrandimage: true
5

I see that you have custom branding for basic auth + openid, but only have openid configured as an auth type.

I believe you need to enable multi auth with the following 2 settings:

opensearch_security.auth.type: ["basicauth","openid"]
opensearch_security.auth.multiple_auth_enabled: true
6

Craig, thanks again for taking the time to look at this one with me. I’ve updated the yml file as you suggested (it looks like it was previously set that way with the line you suggest commented out in favour of the single auth line.) There’s no material change when I try a docker stop/start on the container except the reload time now appears to be 6 seconds so something’s different!

-Adam

7

If I do a docker inspect I find more log items there. I’m seeing “curl: (7) Failed to connect to localhost port 5601 after 0 ms: Couldn’t connect to server” which looks to be my problem I guess…

8

@adam.burgess Did you deploy with docker-compose.yml? If so, could you share that file?
Also please share docker container ls.

Is there any other process using port 5601 on that host?