Dashboards fails to start after restart when DLS role is created

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue:
I’m running into a pretty frustrating issue with OpenSearch when using Docker. After creating a custom role with DLS and setting up role mappings, the OpenSearch Dashboards container fails to start after the cluster is stopped and restarted. The entire cluster becomes unusable until I completely delete everything and start fresh.

Steps to Reproduce

  1. Download the latest official docker-compose file from OpenSearch (example: https://github.com/opensearch-project/opensearch-build/blob/main/docker/release/dockercomposefiles/docker-compose-default.x.yml).

  2. Run the cluster:

    OPENSEARCH_INITIAL_ADMIN_PASSWORD=YourPasswordHere
    docker compose up -d
    
    

    This will start a basic 2-node cluster.

  3. Create a script called test.sh with the following content and run it to create the custom role and role mapping:

    #!/bin/bash
    echo "Creating special_role role"
    curl -X PUT "https://localhost:9200/_plugins/_security/api/roles/special_role?pretty" -H "Content-Type: application/json" -d '{
      "cluster_permissions": ["cluster_composite_ops_ro"],
      "index_permissions": [
        {
          "index_patterns" : ["*"],
          "dls": "{\"bool\":{\"should\":[{\"bool\":{\"must_not\":{\"exists\":{\"field\":\"_acls\"}}}},{\"terms\":{\"_acls.keyword\":[\"${user.name}\", ${user.securityRoles},${user.roles}]}}]}}",
          "allowed_actions": ["read"]
        }
      ],
      "tenant_permissions": []
    }' --insecure -u admin:"${OPENSEARCH_INITIAL_ADMIN_PASSWORD}"
    
    echo "Creating mappings for special_role role"
    curl -X PUT "https://localhost:9200/_plugins/_security/api/rolesmapping/special_role?pretty" \
      -H "Content-Type: application/json" \
      -d '{
          "users": ["*"],
          "backend_roles": [],
          "hosts": []
        }' --insecure -u admin:"${OPENSEARCH_INITIAL_ADMIN_PASSWORD}"
    
    echo "Verifying creation of special_role role..."
    curl -X GET "https://localhost:9200/_plugins/_security/api/roles/special_role" \
      --insecure -u "admin:${OPENSEARCH_INITIAL_ADMIN_PASSWORD}" 
    
    
  4. Stop the OpenSearch cluster using Docker Desktop and restart it.

Problem

Immediately after restart, OpenSearch Dashboards logs the following errors and exits with a fatal failure:

{"type":"log","@timestamp":"2025-12-11T15:08:17Z","tags":["info","savedobjects-service"],"pid":1,"message":"Starting saved objects migrations"}

{"type":"log","@timestamp":"2025-12-11T15:08:17Z","tags":["error","opensearch","data"],"pid":1,"message":"[security_exception]: Error while evaluating DLS/FLS privileges"}

{"type":"log","@timestamp":"2025-12-11T15:08:17Z","tags":["warning","savedobjects-service"],"pid":1,"message":"Unable to connect to OpenSearch. Error: security_exception: [security_exception] Reason: Error while evaluating DLS/FLS privileges"}

{"type":"log","@timestamp":"2025-12-11T15:08:17Z","tags":["fatal","root"],"pid":1,"message":"ResponseError: security_exception: [security_exception] Reason: Error while evaluating DLS/FLS privileges\n    at onBody (/usr/share/opensearch-dashboards/node_modules/@opensearch-project/opensearch/lib/Transport.js:426:23)\n    at IncomingMessage.onEnd (/usr/share/opensearch-dashboards/node_modules/@opensearch-project/opensearch/lib/Transport.js:341:11)\n    at IncomingMessage.emit (node:events:530:35)\n    at endReadableNT (node:internal/streams/readable:1698:12)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21) {\n  meta: {\n    body: { error: [Object], status: 500 },\n    statusCode: 500,\n    headers: {\n      'x-opensearch-version': 'OpenSearch/3.2.0 (opensearch)',\n      'content-type': 'application/json; charset=UTF-8',\n      'content-length': '203'\n    },\n    meta: {\n      context: null,\n      request: [Object],\n      name: 'opensearch-js',\n      connection: [Object],\n      attempts: 0,\n      aborted: false\n    }\n  }\n}"}

{"type":"log","@timestamp":"2025-12-11T15:08:17Z","tags":["info","plugins-system"],"pid":1,"message":"Stopping all plugins."}


 FATAL  {"error":{"root_cause":[{"type":"security_exception","reason":"Error while evaluating DLS/FLS privileges"}],"type":"security_exception","reason":"Error while evaluating DLS/FLS privileges"},"status":500}

Impact

At this point the only fix is to run:

docker compose down -v

…which deletes all data, including any indices or dashboards I’ve created.
This makes it basically impossible to use DLS with Docker without risking data loss.

Question

Is this a known OpenSearch bug?
Is there any workaround that prevents Dashboards from crashing on restart after defining a DLS-enabled role?

Configuration:

Relevant Logs or Screenshots:

@sdn Did you try making changes with securityadmin.sh instead of direct APIs?

@sdn I’ve tested your script. I had no issues after running it and restarting all OpenSearch nodes in the cluster. OpenSearch Dashboards didn’t produce any errors and successfully connected to the OS cluster.

This was tested with version OS 3.3.2 and OSD 3.3.0

Creating special_role role
{
  "status" : "OK",
  "message" : "'special_role' updated."
}
Creating mappings for special_role role
{
  "status" : "OK",
  "message" : "'special_role' updated."
}
Verifying creation of special_role role...
{"special_role":{"reserved":false,"hidden":false,"cluster_permissions":["cluster_composite_ops_ro"],"index_permissions":[{"index_patterns":["*"],"dls":"{\"bool\":{\"should\":[{\"bool\":{\"must_not\":{\"exists\":{\"field\":\"_acls\"}}}},{\"terms\":{\"_acls.keyword\":[\"${user.name}\", ${user.securityRoles},${user.roles}]}}]}}","fls":[],"masked_fields":[],"allowed_actions":["read"]}],"tenant_permissions":[],"static":false}}

@sdn I’ve checked your script again and I’ve found 2 issues.

  1. You’re applying the special_role to all users (roles_mappings)
    This includes admin and kibanaserver users. OpenSearch Dashboards uses kibanaserver user to connect with OpenSearch nodes.

  2. The ‘special_role’ is applied to all indices.
    This includes all the system indices like .kibana* (this holds all saved objects: Dashboards, Visualisations, saved searches, etc.), .opendistro_security (this holds all security configuration and can be accessed directly only with superadmin certificate).

My advice is to narrow down users and indices in both roles and roles_mapping.
Avoid applying any DLS or FLS to any system indices (starting with .)

In my test, I’ve noticed additional errors in OpenSearch nodes.

opensearch-node2       | [2026-02-16T23:49:25,812][ERROR][o.o.s.c.DlsFlsValveImpl  ] [opensearch-node2] Error while evaluating DLS/FLS privileges
opensearch-node2       | org.opensearch.security.privileges.PrivilegesEvaluationException: Invalid DLS query: {"bool":{"should":[{"bool":{"must_not":{"exists":{"field":"_acls"}}}},{"terms":{"_acls.keyword":["kibanaserver", "special_role","own_index","kibana_server",]}}]}}
...
opensearch-node2       | Caused by: org.opensearch.security.privileges.PrivilegesConfigurationValidationException: Invalid DLS query: {"bool":{"should":[{"bool":{"must_not":{"exists":{"field":"_acls"}}}},{"terms":{"_acls.keyword":["kibanaserver", "special_role","own_index","kibana_server",]}}]}}
opensearch-node2       |        at org.opensearch.security.privileges.dlsfls.DocumentPrivileges$DlsQuery.parseQuery(DocumentPrivileges.java:132)
opensearch-node2       |        at org.opensearch.security.privileges.dlsfls.DocumentPrivileges$DlsQuery$Dynamic.evaluate(DocumentPrivileges.java:185)
opensearch-node2       |        ... 74 more
opensearch-node2       | Caused by: org.opensearch.core.xcontent.XContentParseException: [1:157] [bool] failed to parse field [should]

According to those errors, your DLS query builds an invalid DLS query.

Please find more information about DLS in the OpenSearch documentation