Occasionally receiving "401Unauthorized - Response Error" when auto refreshing dashboards

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 1.3.0
OpenSearch Dashboards 1.3.6
Keycloak 11.0.0
OS - Ubuntu Linux 20.04.1

Describe the issue:
We have an OpenSearch solution using Docker containers deployed in a Kubernetes environment. We utilize Keycloak to handle our authentication. We are able to login users and navigate around Dashboards without issue. However, we can across an issue where if the auto-refresh setting is enabled and a user leaves the dashboard alone, after some time (it has varied from 3 min to over an hour) we get “{“statusCode”:401,“error”:“Unauthorized”,“message”:“Response Error”}” returned and only remediation is to refresh the browser. I have noticed in this post [BUG] Kibana throws errors 500/401 one hour after login when using SAML · Issue #828 · opensearch-project/security-dashboards-plugin · GitHub that clearing cookies will cure the issue, however, the environment described is different from ours and want to determine if our issue is related to this one and we just need to wait for a bug fix, or can it be something else. I see errors OpenSearch, OpenSearch Dashboards and Keycloak logs which are provided in below section. However, these logs dont always seem to line up with when the issue is noticed so Im not sure if they are separate issues. But in Dashboards I see a response error stating “certifcate unknown”. In OpenSearch, Ive seen “all shards failed” message. And in Keylcoak Ive seen “refresh token error”.
Can someone assist in confirming that the issue noticed in [BUG] Kibana throws errors 500/401 one hour after login when using SAML · Issue #828 · opensearch-project/security-dashboards-plugin · GitHub is the same as ours or is this a different issue that needs further review? Thank you in advance.

Configuration:

Relevant Logs or Screenshots:
In OpenSearch Dashboard logs
This is logged numerous times even when UI is working fine

{
   "type":"error",
   "@timestamp":"2023-05-19T16:15:44Z",
   "tags":[
      "connection",
      "client",
      "error"
   ],
   "pid":1,
   "level":"error",
   "error":{
      "message":"140574130542400:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46\n",
      "name":"Error",
      "stack":"Error: 140574130542400:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46\n"
   },
   "message":"140574130542400:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46\n"
}

Found in Opensearch logs
An “all shards failed” message was seen in the logs. Will provide the actual log when I get the environment back up and running.

Found in Keycloak logs
Showing that userID is null:
20:04:41,730 WARN [org.keycloak.events] (default task-21) type=REFRESH_TOKEN_ERROR, realmId=d28fe506-8724-4152-8e39-4f45088705f3, clientId=ourClientID, userId=null, ipAddress=<correct IP displayed>, error=invalid_token, grant_type=refresh_token, client_auth_method=client-secret

2

Hi @jpelletier_2023

Could you share the following files?

  • config.yml
  • opensearch_dashboards.yml
  • docker-compose.yml
3

Hi @Eugene7. Thanks for responding. We use helm to configure and deploy our containers so I dont have a docker compose file. Also, Im not sure what the config.yaml file is in reference to. Is this the config file for OpenSearch? If so, I can get that over to you as well. Here is my opensearch-dashboards.yaml:

opensearch:
  hosts:
  -  https://opensearch-hostname.svc.cluster.local:9200

  username: "username"
  password: "password"

  ssl:
    truststore:
      path: /secure/project/keystore/keystore
      password: "password"

  requestHeadersWhitelist:
  - Authorization
  - securitytenant
  # Including old header name for backwards compatability
  - security_tenant

server:
  host: 0.0.0.0

  ssl:
    enabled: true
    keystore:
      path: /secure/project/keystore/keystore
      password: "password"

opensearch_security:
  auth:
    type: openid
  cookie:
    secure: true
    password: "password"

  openid:
    connect_url: "https:<path to keycloak realms>/.well-known/openid-configuration/"
    client_secret: "secret"
    client_id: "id"
    root_ca: /secure/project/oauth-ca/oauth-ca.crt
    base_redirect_url: https://<keycloak IP>

  readonly_mode:
    roles:
    - kibana_read_only

  multitenancy:
    enabled: false

# Increase autocomplete timeout and terminate after so controls visualizations
# are as complete as possible
opensearchDashboards:
  autocompleteTimeout: 100000
  autocompleteTerminateAfter: 10000000

Thank you for taking a look at and let me know if there is anything else you would find helpful to review.