# Authentication finally failed when not using dashboard

**URL:** <https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675>\
**Category:** Security\
**Tags:** troubleshoot\
**Created:** [March 7, 2025, 11:45am UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675 "2025-03-07T11:45:13Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ghata](https://avatars.discourse-cdn.com/v4/letter/g/ecd19e/32.png) [@Ghata](https://forum.opensearch.org/u/Ghata)\
**Post date:** [March 7, 2025, 11:45am UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/1 "2025-03-07T11:45:13Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):  
Opensearch version: 2.31.0

**Describe the issue** :  
when I am using custom security config, getting the error

```auto
Authentication finally failed for null 

```

**Configuration** :  
here’s my values.yaml:

```auto
image:
  repository: "opensearchproject/opensearch"
imagePullSecrets:
  - name: redacted
replicas: 2
persistence:
  enabled: false
config:
  opensearch.yml: |
    plugins:
      security:
        allow_default_init_securityindex: true
        allow_unsafe_democertificates: true
        authcz:
          admin_dn:
            - 'CN=root.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
            - 'CN=A,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
        nodes_dn:
            - 'CN=node1.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
        ssl:
          transport:
            pemcert_filepath: node1-secret
            pemkey_filepath: node1-key
            pemtrustedcas_filepath: root-secret
            enforce_hostname_verification: false
          http:
            enabled: true
            pemcert_filepath: node1-secret
            pemkey_filepath: node1-key
            pemtrustedcas_filepath: root-secret
        audit.type: internal_opensearch
        restapi:
          roles_enabled: ["all_access", "security_rest_api_access"]
        system_indices:
          enabled: true
          indices:
            [
              ".opendistro-alerting-config",
              ".opendistro-alerting-alert*",
              ".opendistro-anomaly-results*",
              ".opendistro-anomaly-detector*",
              ".opendistro-anomaly-checkpoints",
              ".opendistro-anomaly-detection-state",
              ".opendistro-reports-*",
              ".opendistro-notifications-*",
              ".opendistro-notebooks",
              ".opendistro-asynchronous-search-response*",
            ]

securityContext:
  allowPrivilegeEscalation: false
  runAsNonRoot: true
extraEnvs: 
  - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
    value: "password"
  - name: DISABLE_INSTALL_DEMO_CONFIG
    value: 'true'
  - name: DISABLE_PERFORMANCE_ANALYZER_AGENT_CLI
    value: 'true'
secretMounts:
  - name: root-secret
    secretName: node-secret
    path: /usr/share/opensearch/config/root-secret
    subPath: ca.crt
  - name: node1-secret
    secretName: node-secret
    path: /usr/share/opensearch/config/node1-secret
    subPath: tls.crt
  - name: node-key
    secretName: node-secret
    path: /usr/share/opensearch/config/node1-key
    subPath: tls.key
plugins:
  enabled: true
  installList:
  - repository-s3
securityConfig:
  enabled: true
  path: "/usr/share/opensearch/config/opensearch-security"
  actionGroupsSecret:
  configSecret:
  internalUsersSecret:
  rolesSecret:
  rolesMappingSecret:
  tenantsSecret:
  config:
    securityConfigSecret: ""
    dataComplete: true
    data: 
      config.yml: |-
        _meta:
          type: "config"
          config_version: 2
        config:
          dynamic:
            http:
              anonymous_auth_enabled: false
              xff:
                enabled: false
                internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
                remoteIpHeader: 'x-forwarded-for'
            authc:
              basic_internal_auth_domain:
                description: "Authenticate via HTTP Basic"
                http_enabled: true
                transport_enabled: true
                order: 0
                http_authenticator:
                  type: "basic"
                  challenge: false
                authentication_backend:
                  type: "internal"
              openid_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: 1
                http_authenticator:
                  type: openid
                  challenge: false
                  config:
                    openid_connect_idp:
                      enable_ssl: true
                      verify_hostnames: true
                      pemtrustedcas_filepath: "/usr/share/opensearch/config/root-secret"
                    subject_key: preferred_username
                    roles_key: roles
                    openid_connect_url: "https://keycloak-url/auth/realms/dedicated/.well-known/openid-configuration"
                authentication_backend:
                  type: noop
      internal_users.yml: |-
            ---
            # This is the internal user database
            # The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh
    
            _meta:
              type: "internalusers"
              config_version: 2
    
            # Define your internal users here
    
            
        
      roles.yml: |-
            ---
            _meta:
              type: "roles"
              config_version: 2
            
        
      roles_mapping.yml: |-
            ---
            # In this file users, backendroles and hosts can be mapped to Security roles.
            # Permissions for OpenSearch roles are configured in roles.yml
    
            _meta:
              type: "rolesmapping"
              config_version: 2
    
            # Define your roles mapping here
    
            ## Demo roles mapping
    
            all_access:
              reserved: false
              backend_roles:
              - "admin"
              description: "Maps admin to all_access"
    
            own_index:
              reserved: false
              users:
              - "*"
              description: "Allow full access to an index named like the username"
    
            logstash:
              reserved: false
              backend_roles:
              - "logstash"
    
            kibana_user:
              reserved: false
              backend_roles:
              - "kibanauser"
              description: "Maps kibanauser to kibana_user"
    
            readall:
              reserved: false
              backend_roles:
              - "readall"
    
            manage_snapshots:
              reserved: false
              backend_roles:
              - "snapshotrestore"
    
            kibana_server:
              reserved: true
              users:
              - "kibanaserver"
        
      action_groups.yml: |-
        _meta:
          type: "actiongroups"
          config_version: 2
    
      tenants.yml: |-
            ---
            _meta:
              type: "tenants"
              config_version: 2
    
            # Define your tenants here
    
            ## Demo tenants
            admin_tenant:
              reserved: false
              description: "Demo tenant for admin user"                
ingress:
    enabled: true
    ingressClassName: nginx
    annotations: 
      nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
      nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    hosts:
      - engine.url
    path: /
    tls: 
      - secretName: tls 
        hosts:
          - engine.url

```

Note: the cert used is self-signed certs generated following documentation:

> **[Generating self-signed certificates](https://opensearch.org/docs/latest/security/configuration/generate-certificates/)**
>
> Generating self-signed certificates

If I disable the securityConfig, then it is working with admin username and password but enabling it fails.

also output of command

```auto
curl -XGET https://localhost:9200 -u 'admin:admin' --insecure
Authentication finally failed

```

**Relevant Logs or Screenshots** :

---

<div class="post-metadata">

**Author:** ![Mantas](https://avatars.discourse-cdn.com/v4/letter/m/7bcc69/32.png) [@Mantas](https://forum.opensearch.org/u/Mantas)\
**Post date:** [March 7, 2025, 11:49am UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/2 "2025-03-07T11:49:14Z")

</div>

> [@Ghata](#):
>
> ```auto
> extraEnvs: 
> - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
> value: "password"
> 
> ```

Hi @Ghata,

What do you have set as a (please do not share) `OPENSEARCH_INITIAL_ADMIN_PASSWORD`, I do not think it is “admin”, is it?

Best,  
mj

---

<div class="post-metadata">

**Author:** ![Ghata](https://avatars.discourse-cdn.com/v4/letter/g/ecd19e/32.png) [@Ghata](https://forum.opensearch.org/u/Ghata)\
**Post date:** [March 7, 2025, 1:00pm UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/3 "2025-03-07T13:00:12Z")

</div>

Hi @Mantas ,

Yes it’s not admin, also it’s not password as well… I have removed the actual password with just the word “password”.

---

<div class="post-metadata">

**Author:** ![Mantas](https://avatars.discourse-cdn.com/v4/letter/m/7bcc69/32.png) [@Mantas](https://forum.opensearch.org/u/Mantas)\
**Post date:** [March 7, 2025, 1:03pm UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/4 "2025-03-07T13:03:28Z")

</div>

> [@Ghata](#):
>
> `curl -XGET https://localhost:9200 -u 'admin:admin' --insecure`

The password you are using in here is the value of `OPENSEARCH_INITIAL_ADMIN_PASSWORD` ?

`curl -XGET https://localhost:9200 -u 'admin:<OPENSEARCH_INITIAL_ADMIN_PASSWORD>' --insecure`

---

<div class="post-metadata">

**Author:** ![Ghata](https://avatars.discourse-cdn.com/v4/letter/g/ecd19e/32.png) [@Ghata](https://forum.opensearch.org/u/Ghata)\
**Post date:** [March 7, 2025, 1:14pm UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/5 "2025-03-07T13:14:47Z")

</div>

Hi @Mantas

Yes I tried with both passwords. Default password for admin user is admin so I tried with

curl -XGET [https://localhost:9200](https://localhost:9200) -u ‘admin:admin’ --insecure as well

and the password I set in variable OPENSEARCH\_INITIAL\_ADMIN\_PASSWORD as well

curl -XGET [https://localhost:9200](https://localhost:9200) -u ‘admin:OPENSEARCH\_INITIAL\_ADMIN\_PASSWORD’ --insecure as well

But both are failing

---

<div class="post-metadata">

**Author:** ![Ghata](https://avatars.discourse-cdn.com/v4/letter/g/ecd19e/32.png) [@Ghata](https://forum.opensearch.org/u/Ghata)\
**Post date:** [March 10, 2025, 6:59am UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/6 "2025-03-10T06:59:15Z")

</div>

Hi @Mantas ,

I executed the curl with -v enabled and there’s a certificate error that I am facing.

```auto
Note: Unnecessary use of -X or --request, GET is already inferred.
* Host opensearch-cluster-master:9200 was resolved.
* IPv6: (none)
* IPv4: 172.20.199.203
* Trying 172.20.199.203:9200...
* Connected to opensearch-cluster-master (172.20.199.203) port 9200
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / RSASSA-PSS
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
* subject: C=CA; ST=ONTARIO; L=TORONTO; O=ORG; OU=UNIT; CN=node1.dns.a-record
* start date: Mar 6 06:05:48 2025 GMT
* expire date: Mar 6 06:05:48 2027 GMT
* issuer: C=CA; ST=ONTARIO; L=TORONTO; O=ORG; OU=UNIT; CN=root.dns.a-record
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/1.x
* Server auth using Basic with user 'admin'
> GET / HTTP/1.1
> Host: opensearch-cluster-master:9200
> Authorization: Basic redacted
> User-Agent: curl/8.5.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/1.1 401 Unauthorized
< content-type: text/plain; charset=UTF-8
< content-length: 29
< 
* Connection #0 to host opensearch-cluster-master left intact
Authentication finally failed

```

Could you suggest what is wrong here?

---

<div class="post-metadata">

**Author:** ![Mantas](https://avatars.discourse-cdn.com/v4/letter/m/7bcc69/32.png) [@Mantas](https://forum.opensearch.org/u/Mantas)\
**Post date:** [March 10, 2025, 10:37am UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/7 "2025-03-10T10:37:53Z")

</div>

Hi @Ghata,

I’ve just noticed you have disabled demo setup and your `internal_users.yml` and `roles.yml` files are empty.

> [@Ghata](#):
>
> ```auto
> - name: DISABLE_INSTALL_DEMO_CONFIG
> value: 'true'
> 
> ```

> [@Ghata](#):
>
> ```auto
> internal_users.yml: |-
> ---
> # This is the internal user database
> # The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh
>     
> _meta:
> type: "internalusers"
> config_version: 2
>     
> # Define your internal users here
>     
>             
>         
> roles.yml: |-
> ---
> _meta:
> type: "roles"
> config_version: 2
>             
>         
> 
> ```

You will need to specify users and the roles in your configuration (you could run with demo setup and copy `internal_users.yml`, `roles.yml`).

here are some samples:

> **[Modifying the YAML files](https://opensearch.org/docs/latest/security/configuration/yaml/#internal_usersyml)**
>
> Modifying the YAML files

> **[Modifying the YAML files](https://opensearch.org/docs/latest/security/configuration/yaml/#rolesyml)**
>
> Modifying the YAML files

best,  
mj

---

<div class="post-metadata">

**Author:** ![Ghata](https://avatars.discourse-cdn.com/v4/letter/g/ecd19e/32.png) [@Ghata](https://forum.opensearch.org/u/Ghata)\
**Post date:** [March 10, 2025, 11:32am UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/8 "2025-03-10T11:32:14Z")

</div>

Hi @Mantas

Thanks for your reply, adding the default roles.yaml and internal\_users.yaml worked.

I have few questions here:

1. the hashed password for the admin internal user should be the value OPENSEARCH\_INITIAL\_ADMIN\_PASSWORD ?
2. If you see the config.yaml,

```auto
config:
          dynamic:
            http:
              anonymous_auth_enabled: false
              xff:
                enabled: false
                internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
                remoteIpHeader: 'x-forwarded-for'
            authc:
              basic_internal_auth_domain:
                description: "Authenticate via HTTP Basic"
                http_enabled: true
                transport_enabled: true
                order: 0
                http_authenticator:
                  type: "basic"
                  challenge: false
                authentication_backend:
                  type: "internal"
              openid_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: 1
                http_authenticator:
                  type: openid
                  challenge: false
                  config:
                    openid_connect_idp:
                      enable_ssl: true
                      verify_hostnames: true
                      pemtrustedcas_filepath: "/usr/share/opensearch/config/root-secret"
                    subject_key: preferred_username
                    roles_key: roles
                    openid_connect_url: "https://keycloak-url/auth/realms/dedicated/.well-known/openid-configuration"
                authentication_backend:
                  type: noop

```

I am integrating it with keycloak so that I am able to access opensearch with jwt as well. However, when I generate the token and try hitting, it is throwing error:

```auto
[2025-03-10T11:23:15,661][WARN][c.a.d.a.h.j.k.SelfRefreshingKeySet] [opensearch-cluster-master-0] KeySetProvider threw error
com.amazon.dlic.auth.http.jwt.keybyoidc.AuthenticatorUnavailableException: Error while getting https://keycloak-url/auth/realms/dedicated/.well-known/openid-configuration: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.getJwksUri(KeySetRetriever.java:167) ~[opensearch-security-2.19.0.0.jar:2.19.0.0]
at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.get(KeySetRetriever.java:72) ~[opensearch-security-2.19.0.0.jar:2.19.0.0]
at com.amazon.dlic.auth.http.jwt.keybyoidc.SelfRefreshingKeySet$1.run(SelfRefreshingKeySet.java:213) [opensearch-security-2.19.0.0.jar:2.19.0.0]

```

I am suspecting the pemtrustedcas\_filepath value is causing a problem here?  
FYI - I have passed root cert which I generated using the self-signed certificate.

1. Also in the admin\_dn of security plugin, why is it needed to pass the admin dns entry in it along with root dns?

```auto
security:
        allow_default_init_securityindex: true
        allow_unsafe_democertificates: true
        authcz:
          admin_dn:
            - 'CN=root.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
            - 'CN=A,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
        nodes_dn:
            - 'CN=node1.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'

```

How does this certificate logic works for the node? If I don’t pass the DN entry of the admin cert, I was getting the error:

```auto
[2025-03-05T10:40:31,881][ERROR][o.o.t.n.s.SecureNetty4Transport] [opensearch-cluster-master-0] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

```

1. For tls communicationbetween opensearch and opensearch dashboard, will the same certificate work ?

---

<div class="post-metadata">

**Author:** ![Mantas](https://avatars.discourse-cdn.com/v4/letter/m/7bcc69/32.png) [@Mantas](https://forum.opensearch.org/u/Mantas)\
**Post date:** [March 11, 2025, 10:35am UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/9 "2025-03-11T10:35:05Z")

</div>

Hi @Ghata,

I suggest keeping your admin password the same; otherwise, once the configuration is applied from your config files (via `securityadmin.sh`), it will overwrite it.

By the looks, you are using your root CA for Keycloak SSL, you need to use your Keycloak certificate, if self-generated, make sure FQDN (and CN) is in SAN.  
i.e.:

```auto
... -subj "/CN=keycloak.test.local" -addext "subjectAltName=DNS:keycloak.test.local, IP: 0.0.0...

```

> **[OpenID Connect](https://opensearch.org/docs/latest/security/authentication-backends/openid-connect/#tls-settings)**
>
> OpenID Connect

You do not need the root DN just the admin DN for your super admin privileges.

For the TLS communication between OpenSearch and OpenSearch Dashboard, you need a client cert signed by the same root CA as your node certs, or you will need to concatenate it with the node root CA to make a chain.

> **[Configure TLS](https://opensearch.org/docs/latest/install-and-configure/install-dashboards/tls/)**
>
> Configure TLS for OpenSearch Dashboards

Hope I covered all of your questions.

Best,  
mj

---

<div class="post-metadata">

**Author:** ![Ghata](https://avatars.discourse-cdn.com/v4/letter/g/ecd19e/32.png) [@Ghata](https://forum.opensearch.org/u/Ghata)\
**Post date:** [March 11, 2025, 2:17pm UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/10 "2025-03-11T14:17:48Z")

</div>

Hi @Mantas ,

Thanks for answering my questions, it helps.

I am still facin issue when trying to communicate with opensearch using jwt token. I am passing the client cert of keycloak as part of the config

```auto
pemtrustedcas_content: |

```

but still getting the same error when I am trying to generate token for opensearch client created in keycloak and hit opensearch engine api:

```auto
[2025-03-10T11:23:15,661][WARN][c.a.d.a.h.j.k.SelfRefreshingKeySet] [opensearch-cluster-master-0] KeySetProvider threw error
com.amazon.dlic.auth.http.jwt.keybyoidc.AuthenticatorUnavailableException: Error while getting https://keycloak-url/auth/realms/dedicated/.well-known/openid-configuration: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.getJwksUri(KeySetRetriever.java:167) ~[opensearch-security-2.19.0.0.jar:2.19.0.0]
at com.amazon.dlic.auth.http.jwt.keybyoidc.KeySetRetriever.get(KeySetRetriever.java:72) ~[opensearch-security-2.19.0.0.jar:2.19.0.0]
at com.amazon.dlic.auth.http.jwt.keybyoidc.SelfRefreshingKeySet$1.run(SelfRefreshingKeySet.java:213) [opensearch-security-2.19.0.0.jar:2.19.0.0]

```

what could be the reason?

---

<div class="post-metadata">

**Author:** ![Mantas](https://avatars.discourse-cdn.com/v4/letter/m/7bcc69/32.png) [@Mantas](https://forum.opensearch.org/u/Mantas)\
**Post date:** [March 18, 2025, 3:45pm UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/11 "2025-03-18T15:45:46Z")

</div>

> [@Ghata](#):
>
> `PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target`

Hi @Ghata,

The above indicates that Keycloak does not like the cert presented.  
Did you get the cert from Keycloak?

best,  
mj

---

<div class="post-metadata">

**Author:** ![Ghata](https://avatars.discourse-cdn.com/v4/letter/g/ecd19e/32.png) [@Ghata](https://forum.opensearch.org/u/Ghata)\
**Post date:** [March 19, 2025, 11:44am UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/12 "2025-03-19T11:44:48Z")

</div>

Hi @Mantas ,  
resolving 1 error after the other, I have reached a state where I am able to generate token using keycloak and hitting engine api to create index. However, on opensearch terminal, when I use the curl command, I get:

curl --insecure -u admin:password -XGET [https://localhost:9200](https://localhost:9200)

```auto
Authentication finally failed for null from

```

I spined up opensearch dashboards with basic configuration and I am unable to curl from dashboard pod as well.

---

<div class="post-metadata">

**Author:** ![Ghata](https://avatars.discourse-cdn.com/v4/letter/g/ecd19e/32.png) [@Ghata](https://forum.opensearch.org/u/Ghata)\
**Post date:** [March 19, 2025, 11:48am UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/13 "2025-03-19T11:48:12Z")

</div>

Hi @Mantas

Here’s my configuration:

```auto
image:
  repository: "opensearchproject/opensearch"
imagePullSecrets:
  - name: xyz
replicas: 2
config:
  opensearch.yml: |
    plugins:
      security:
        allow_default_init_securityindex: true
        allow_unsafe_democertificates: false
        authcz:
          admin_dn:
            - 'CN=A,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
        nodes_dn: 
          - 'CN=node1.dns.a-record,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
        ssl:
          transport:
            pemcert_filepath: node1-secret
            pemkey_filepath: node1-key
            pemtrustedcas_filepath: root-secret
            enforce_hostname_verification: false
            #resolve_hostname: false
          http:
            enabled: true
            pemcert_filepath: node1-secret
            pemkey_filepath: node1-key
            pemtrustedcas_filepath: root-secret
        audit.type: internal_opensearch
        restapi:
          roles_enabled: ["all_access", "security_rest_api_access"]
        system_indices:
          enabled: true
          indices:
            [
              ".opendistro-alerting-config",
              ".opendistro-alerting-alert*",
              ".opendistro-anomaly-results*",
              ".opendistro-anomaly-detector*",
              ".opendistro-anomaly-checkpoints",
              ".opendistro-anomaly-detection-state",
              ".opendistro-reports-*",
              ".opendistro-notifications-*",
              ".opendistro-notebooks",
              ".opendistro-asynchronous-search-response*",
            ]

securityContext:
  allowPrivilegeEscalation: false
  runAsNonRoot: true
extraEnvs: 
  - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
    value: password
  - name: DISABLE_INSTALL_DEMO_CONFIG
    value: 'true'
  - name: DISABLE_PERFORMANCE_ANALYZER_AGENT_CLI
    value: 'true'
secretMounts:
  - name: root-secret
    secretName: node-secret
    path: /usr/share/opensearch/config/root-secret
    subPath: ca.crt
  - name: node1-secret
    secretName: node-secret
    path: /usr/share/opensearch/config/node1-secret
    subPath: tls.crt
  - name: node-key
    secretName: node-secret
    path: /usr/share/opensearch/config/node1-key
    subPath: tls.key
plugins:
  enabled: true
  installList:
  - repository-azure
securityConfig:
  enabled: true
  path: "/usr/share/opensearch/config/opensearch-security"
  actionGroupsSecret:
  configSecret:
  internalUsersSecret:
  rolesSecret:
  rolesMappingSecret:
  tenantsSecret:
  config:
    securityConfigSecret: ""
    dataComplete: true
    data: 
      config.yml: |-
        _meta:
          type: "config"
          config_version: 2
        config:
          dynamic:
            http:
              anonymous_auth_enabled: false
            authc:
              basic_internal_auth_domain:
                description: "Authenticate via HTTP Basic"
                http_enabled: true
                transport_enabled: true
                order: 0
                http_authenticator:
                  type: "basic"
                  challenge: false
                authentication_backend:
                  type: "internal"
              openid_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: 1
                http_authenticator:
                  type: openid
                  challenge: false
                  config:
                    openid_connect_idp:
                      enable_ssl: true
                      verify_hostnames: true
                      #pemtrustedcas_filepath: "/usr/share/opensearch/config/root-secret"
                      pemtrustedcas_content: |
                        -----BEGIN CERTIFICATE-----
                        -----END CERTIFICATE-----
                    subject_key: preferred_username
                    roles_key: roles
                    openid_connect_url: "https://keycloak-url/auth/realms/tenant/.well-known/openid-configuration"
                authentication_backend:
                  type: noop
      internal_users.yml: |-
            ---
            # This is the internal user database
            # The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh
    
            _meta:
              type: "internalusers"
              config_version: 2
            

            ## Demo users

            admin:
              hash: "$2a$12$VcCDgh2NDk07JGN0rjGbM.Ad41qVR/YFJcgHp0UGns5JDymv..TOG"
              reserved: true
              backend_roles:
              - "admin"
              description: "Demo admin user"            
    
            
        
      roles.yml: |-
            ---
            _meta:
              type: "roles"
              config_version: 2
            service_access:
              reserved: false
              hidden: false
              cluster_permissions:
              - "cluster:admin/opendistro/alerting/alerts/ack"
              - "cluster:admin/opendistro/alerting/alerts/get"
              - "cluster:monitor/*"
              - "indices:data/read/*"
              index_permissions:
              - index_patterns:
                - "*"
                allowed_actions:
                - "indices:admin/template/get"
                - "indices:admin/template/put"
                - "indices:data/read/*"
                - "indices:monitor/*"
                - "indices:admin/get"
                - "indices:admin/mappings/get"
                - "indices:admin/aliases/get"   
                - "indices:admin/create"                  
        
      roles_mapping.yml: |-
            ---
            # In this file users, backendroles and hosts can be mapped to Security roles.
            # Permissions for OpenSearch roles are configured in roles.yml
    
            _meta:
              type: "rolesmapping"
              config_version: 2
    
            # Define your roles mapping here
    
            ## Demo roles mapping
    
            all_access:
              reserved: true
              backend_roles:
              - "admin"
              description: "Maps admin to all_access"
            
            service_access:
              reserved: false
              hidden: false
              hosts: []
              users:
              - "*"
            
            own_index:
              reserved: false
              users:
              - "*"
              description: "Allow full access to an index named like the username"
    
        
      action_groups.yml: |-
        _meta:
          type: "actiongroups"
          config_version: 2
    
      tenants.yml: |-
            ---
            _meta:
              type: "tenants"
              config_version: 2
    
            # Define your tenants here
    
                            
ingress:
    enabled: true
    ingressClassName: nginx
    annotations: 
      nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
      nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    hosts:
      - engine.url
    path: /
    tls: 
      - secretName: secretsan
        hosts:
          - engine.url

```

---

<div class="post-metadata">

**Author:** ![Mantas](https://avatars.discourse-cdn.com/v4/letter/m/7bcc69/32.png) [@Mantas](https://forum.opensearch.org/u/Mantas)\
**Post date:** [March 19, 2025, 12:58pm UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/14 "2025-03-19T12:58:55Z")

</div>

> [@Ghata](#):
>
> curl --insecure -u admin:password -XGET [https://localhost:9200](https://localhost:9200)
> 
> `Authentication finally failed for null from`

This suggests the incorrect credentials, I would guess it is the password as the admin user seems to be existing.

> [@Ghata](#):
>
> ```auto
> admin:
> hash: "$2a$12$VcCDgh2NDk07JGN0rjGbM.Ad41qVR/YFJcgHp0UGns5JDymv..TOG"
> reserved: true
> backend_roles:
> - "admin"
> description: "Demo admin user"            
>     
> 
> ```

Moreover, the all\_access role is not defined in your roles.yml, tho I see all users are mapped to `service_access`..

> [@Ghata](#):
>
> ```auto
> all_access:
> reserved: true
> backend_roles:
> - "admin"
> description: "Maps admin to all_access"
> 
> ```

> [@Ghata](#):
>
> ```auto
> roles.yml: |-
> ---
> _meta:
> type: "roles"
> config_version: 2
> service_access:
> reserved: false
> hidden: false
> cluster_permissions:
> - "cluster:admin/opendistro/alerting/alerts/ack"
> - "cluster:admin/opendistro/alerting/alerts/get"
> - "cluster:monitor/*"
> - "indices:data/read/*"
> index_permissions:
> - index_patterns:
> - "*"
> allowed_actions:
> - "indices:admin/template/get"
> - "indices:admin/template/put"
> - "indices:data/read/*"
> - "indices:monitor/*"
> - "indices:admin/get"
> - "indices:admin/mappings/get"
> - "indices:admin/aliases/get"   
> - "indices:admin/create"                  
> 
> ```

---

<div class="post-metadata">

**Author:** ![Ghata](https://avatars.discourse-cdn.com/v4/letter/g/ecd19e/32.png) [@Ghata](https://forum.opensearch.org/u/Ghata)\
**Post date:** [March 19, 2025, 4:10pm UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/15 "2025-03-19T16:10:34Z")

</div>

Hi @Mantas  
2 questions here

if I am setting the admin password via OPENSEARCH\_INITIAL\_ADMIN\_PASSWORD env variable then in internal users admin user is not a must to set?

the all\_access role mapping is defined but not the role because I copied default configuration from open search documents. So, what roles, role mapping is must to be defined?

---

<div class="post-metadata">

**Author:** ![Mantas](https://avatars.discourse-cdn.com/v4/letter/m/7bcc69/32.png) [@Mantas](https://forum.opensearch.org/u/Mantas)\
**Post date:** [March 19, 2025, 4:18pm UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/16 "2025-03-19T16:18:50Z")

</div>

> [@Ghata](#):
>
> if I am setting the admin password via OPENSEARCH\_INITIAL\_ADMIN\_PASSWORD env variable then in internal users admin user is not a must to set?

Let’s address question #1 first and we will get back to the other one later.

can you make sure that the hash value is the same as OPENSEARCH\_INITIAL\_ADMIN\_PASSWORD env:

I suspect you are overwriting the password:

> [@Ghata](#):
>
> ```auto
> admin:
> hash: "$2a$12$VcCDgh2NDk07JGN0rjGbM.Ad41qVR/YFJcgHp0UGns5JDymv..TOG"
> 
> ```

To generate one, run `plugins/opensearch-security/tools/hash.sh -p <password as per OPENSEARCH_INITIAL_ADMIN_PASSWORD env>`

Best,  
mj

---

<div class="post-metadata">

**Author:** ![Ghata](https://avatars.discourse-cdn.com/v4/letter/g/ecd19e/32.png) [@Ghata](https://forum.opensearch.org/u/Ghata)\
**Post date:** [March 20, 2025, 6:52am UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/18 "2025-03-20T06:52:52Z")

</div>

Hi @Mantas ,

Yes that worked.  
Appreciate all your support on this.

Could you please help on my second query?

Regards,

---

<div class="post-metadata">

**Author:** ![Mantas](https://avatars.discourse-cdn.com/v4/letter/m/7bcc69/32.png) [@Mantas](https://forum.opensearch.org/u/Mantas)\
**Post date:** [March 20, 2025, 12:20pm UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/19 "2025-03-20T12:20:56Z")

</div>

Could you share the outputs of the below:

```auto

curl --insecure -u admin:password -XGET "http://localhost:9200/_plugins/_security/api/roles?pretty"

```

```auto

curl --insecure -u admin:password -XGET "https://localhost:9200/_plugins/_security/authinfo?pretty"

```

best,  
mj

---

<div class="post-metadata">

**Author:** ![Ghata](https://avatars.discourse-cdn.com/v4/letter/g/ecd19e/32.png) [@Ghata](https://forum.opensearch.org/u/Ghata)\
**Post date:** [March 21, 2025, 12:51pm UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/20 "2025-03-21T12:51:19Z")

</div>

> [@Mantas](#):
>
> `_plugins/_security/api/roles?pretty`

@Mantas ,

Here’s the requested output:  
curl --insecure -u admin:password -XGET “[http://localhost:9200/\_plugins/\_security/api/roles?pretty](http://localhost:9200/_plugins/_security/api/roles?pretty)”

```auto
{
  "manage_snapshots" : {
    "reserved" : true,
    "hidden" : false,
    "description" : "Provide the minimum permissions for managing snapshots",
    "cluster_permissions" : [
      "manage_snapshots"
    ],
    "index_permissions" : [
      {
        "index_patterns" : [
          "*"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "indices:data/write/index",
          "indices:admin/create"
        ]
      }
    ],
    "tenant_permissions" : [],
    "static" : true
  },
  "logstash" : {
    "reserved" : true,
    "hidden" : false,
    "description" : "Provide the minimum permissions for logstash and beats",
    "cluster_permissions" : [
      "cluster_monitor",
      "cluster_composite_ops",
      "indices:admin/template/get",
      "indices:admin/template/put",
      "cluster:admin/ingest/pipeline/put",
      "cluster:admin/ingest/pipeline/get"
    ],
    "index_permissions" : [
      {
        "index_patterns" : [
          "logstash-*",
          "ecs-logstash-*"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "crud",
          "create_index"
        ]
      },
      {
        "index_patterns" : [
          "*beat*"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "crud",
          "create_index"
        ]
      }
    ],
    "tenant_permissions" : [],
    "static" : true
  },
  "readall_and_monitor" : {
    "reserved" : true,
    "hidden" : false,
    "description" : "Provide the minimum permissions for to readall indices and monitor the cluster",
    "cluster_permissions" : [
      "cluster_monitor",
      "cluster_composite_ops_ro"
    ],
    "index_permissions" : [
      {
        "index_patterns" : [
          "*"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "read"
        ]
      }
    ],
    "tenant_permissions" : [],
    "static" : true
  },
  "service_access" : {
    "reserved" : false,
    "hidden" : false,
    "cluster_permissions" : [
      "cluster:admin/opendistro/alerting/alerts/ack",
      "cluster:admin/opendistro/alerting/alerts/get",
      "cluster:monitor/*",
      "indices:data/read/*"
    ],
    "index_permissions" : [
      {
        "index_patterns" : [
          "*"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "indices:admin/template/get",
          "indices:admin/template/put",
          "indices:data/read/*",
          "indices:monitor/*",
          "indices:admin/get",
          "indices:admin/mappings/get",
          "indices:admin/aliases/get",
          "indices:admin/create"
        ]
      }
    ],
    "tenant_permissions" : [],
    "static" : false
  },
  "kibana_user" : {
    "reserved" : true,
    "hidden" : false,
    "description" : "Provide the minimum permissions for a kibana user",
    "cluster_permissions" : [
      "cluster_composite_ops"
    ],
    "index_permissions" : [
      {
        "index_patterns" : [
          ".kibana",
          ".kibana-6",
          ".kibana_*",
          ".opensearch_dashboards",
          ".opensearch_dashboards-6",
          ".opensearch_dashboards_*"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "read",
          "delete",
          "manage",
          "index"
        ]
      },
      {
        "index_patterns" : [
          ".tasks",
          ".management-beats",
          "*:.tasks",
          "*:.management-beats"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "indices_all"
        ]
      }
    ],
    "tenant_permissions" : [],
    "static" : true
  },
  "own_index" : {
    "reserved" : true,
    "hidden" : false,
    "description" : "Allow all for indices named like the current user",
    "cluster_permissions" : [
      "cluster_composite_ops"
    ],
    "index_permissions" : [
      {
        "index_patterns" : [
          "${user_name}"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "indices_all"
        ]
      }
    ],
    "tenant_permissions" : [],
    "static" : true
  },
  "all_access" : {
    "reserved" : true,
    "hidden" : false,
    "description" : "Allow full access to all indices and all cluster APIs",
    "cluster_permissions" : [
      "*"
    ],
    "index_permissions" : [
      {
        "index_patterns" : [
          "*"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "*"
        ]
      }
    ],
    "tenant_permissions" : [
      {
        "tenant_patterns" : [
          "*"
        ],
        "allowed_actions" : [
          "kibana_all_write"
        ]
      }
    ],
    "static" : true
  },
  "kibana_server" : {
    "reserved" : true,
    "hidden" : false,
    "description" : "Provide the minimum permissions for the Kibana server",
    "cluster_permissions" : [
      "cluster_monitor",
      "cluster_composite_ops",
      "manage_point_in_time",
      "indices:admin/template*",
      "indices:admin/index_template*",
      "indices:data/read/scroll*"
    ],
    "index_permissions" : [
      {
        "index_patterns" : [
          ".kibana",
          ".opensearch_dashboards"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "indices_all"
        ]
      },
      {
        "index_patterns" : [
          ".kibana-6",
          ".opensearch_dashboards-6"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "indices_all"
        ]
      },
      {
        "index_patterns" : [
          ".kibana_*",
          ".opensearch_dashboards_*"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "indices_all"
        ]
      },
      {
        "index_patterns" : [
          ".tasks"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "indices_all"
        ]
      },
      {
        "index_patterns" : [
          ".management-beats*"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "indices_all"
        ]
      },
      {
        "index_patterns" : [
          "*"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "indices:admin/aliases*"
        ]
      }
    ],
    "tenant_permissions" : [],
    "static" : true
  },
  "readall" : {
    "reserved" : true,
    "hidden" : false,
    "description" : "Provide the minimum permissions for to readall indices",
    "cluster_permissions" : [
      "cluster_composite_ops_ro"
    ],
    "index_permissions" : [
      {
        "index_patterns" : [
          "*"
        ],
        "fls" : [],
        "masked_fields" : [],
        "allowed_actions" : [
          "read"
        ]
      }
    ],
    "tenant_permissions" : [],
    "static" : true
  }
}

```

curl --insecure -u admin:password -XGET “[https://localhost:9200/\_plugins/\_security/authinfo?pretty](https://localhost:9200/_plugins/_security/authinfo?pretty)”

```auto
{
  "user" : "User [name=admin, backend_roles=[admin], requestedTenant=null]",
  "user_name" : "admin",
  "user_requested_tenant" : null,
  "remote_address" : "x.x.x.x:y",
  "backend_roles" : [
    "admin"
  ],
  "custom_attribute_names" : [],
  "roles" : [
    "service_access",
    "own_index",
    "all_access"
  ],
  "tenants" : {
    "global_tenant" : true,
    "admin" : true
  },
  "principal" : null,
  "peer_certificates" : "0",
  "sso_logout_url" : null
}

```

---

<div class="post-metadata">

**Author:** ![Mantas](https://avatars.discourse-cdn.com/v4/letter/m/7bcc69/32.png) [@Mantas](https://forum.opensearch.org/u/Mantas)\
**Post date:** [March 25, 2025, 10:34am UTC](https://forum.opensearch.org/t/authentication-finally-failed-when-not-using-dashboard/23675/21 "2025-03-25T10:34:03Z")

</div>

> [@Ghata](#):
>
> the all\_access role mapping is defined but not the role because I copied default configuration from open search documents. So, what roles, role mapping is must to be defined?

You are correct the `all_access` role is defined and mapped as expected.

> [@Ghata](#):
>
> ```auto
> "roles" : [
> "service_access",
> "own_index",
> "all_access"
> ],
> 
> ```

> [@Ghata](#):
>
> ```auto
> "all_access" : {
> "reserved" : true,
> "hidden" : false,
> "description" : "Allow full access to all indices and all cluster APIs",
> "cluster_permissions" : [
> "*"
> ],
> "index_permissions" : [
> {
> "index_patterns" : [
> "*"
> ],
> "fls" : [],
> "masked_fields" : [],
> "allowed_actions" : [
> "*"
> ]
> }
> ],
> "tenant_permissions" : [
> {
> "tenant_patterns" : [
> "*"
> ],
> "allowed_actions" : [
> "kibana_all_write"
> ]
> }
> ],
> "static" : true
> },
> 
> ```

best,  
mj