Authentication finally failed for anonymous user over http randomly

manoj · May 10, 2022, 4:57am

Hi,
I tried to deploy Opensearch 1.2.4 with Opensearch Security Plugin 1.2.4.0 with TLS enabled for transport and TLS disabled for http.
Once started, when I tried to hit localhost:9200, it says “Authentication finally failed”.

$ curl localhost:9200
Authentication finally failed

opensearch.yml
...
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.pemcert_filepath: "srvcert.pem"
plugins.security.ssl.transport.pemkey_filepath: "srvprivkey.pem"
plugins.security.ssl.transport.pemtrustedcas_filepath: "client-cacertbundle.pem"
plugins.security.ssl.http.enabled: false
plugins.security.nodes_dn:
  - CN=localhost
plugins.security.authcz.admin_dn:
  - CN=localhost
plugins.security.allow_default_init_securityindex: true
plugins.security.allow_unsafe_democertificates: false
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: false
...

config.yml
...
config:
  dynamic:
    http:
      anonymous_auth_enabled: true
...

roles.yml
...
anonymous_admin:
  reserved: false
  cluster_permissions:
    - '*'
  index_permissions:
    - index_patterns:
        - '*'
      allowed_actions:
        - '*'
...

roles_mapping.yml
...
 anonymous_admin:
   reserved: false
   backend_roles:
   - "anonymous_admin"
   - "admin"
   users:
   - "opendistro_security_anonymous"
...

Is my setup correct ?

Anthony · May 10, 2022, 8:22am

@manoj You need to specify the username and password, Assuming basic_auth is switched on in config.yaml.

try:

curl -u admin:admin -X GET http://localhost:9200

manoj · May 10, 2022, 8:30am

Hi @Anthony ,
I’m enabling anonymous_auth to omit all type of authentications including basic authentication (i.e., username and password).

config.yml
...
config:
  dynamic:
    http:
      anonymous_auth_enabled: true
...

$ curl -u admin:admin -X GET http://localhost:9200
Authentication finally failed
$ curl -X GET http://localhost:9200
Authentication finally failed

Sometimes, it’s working fine (i.e., if I redeploy/reinstall again and again, it works few times).
But most of the times it fails.

curl localhost:9200
{
“name” : “searchengine”,
“cluster_name” : “searchengine”,
“cluster_uuid” : “ZeWv2Nx_SlGm-nZzzfUC1A”,
“version” : {
“number” : “7.10.2”,
“build_type” : “tar”,
“build_hash” : “4b2cb1b16f50ce4c9bd324d4b1a3274894b59a00”,
“build_date” : “2022-02-02T05:17:09.519459Z”,
“build_snapshot” : false,
“lucene_version” : “8.10.1”,
“minimum_wire_compatibility_version” : “6.8.0”,
“minimum_index_compatibility_version” : “6.0.0-beta1”
},
“tagline” : “The OpenSearch Project: https://opensearch.org/”
}

Anthony · May 10, 2022, 9:01am

@manoj The anonymous user is received with backend role: “opendistro_security_anonymous_backendrole”

Therefore to get this to work I created the following:

roles_mapping.yml:

anonymous:
  backend_roles:
    - "opendistro_security_anonymous_backendrole"

roles.yaml:

anonymous:
  cluster_permissions:
  - "cluster_composite_ops"
  - "cluster:monitor/main"
  index_permissions:
    - index_patterns:
      - "*"
      allowed_actions:
      - "read"

manoj · May 10, 2022, 11:53am

@Anthony .
It tried the same now.
It’s still failing.

Anthony · May 10, 2022, 12:29pm

@manoj How are you deploying? Binary, docker?

Did you reload the security config using securityadmin.sh script after the changes in yaml files?

manoj · May 10, 2022, 12:40pm

@Anthony ,
I’m building docker image with OpenSUSE as BaseOS.
And using Opensearch.tar.gz inside the docker image to run the opensearch.
For every testing I’m running/deploying the docker image as new deployment.

Anthony · May 10, 2022, 2:00pm

@manoj I would recommend to connect to the container running opensearch and use securityadmin.sh script to extract current config using --retrieve option (using admin certificates) and examine the config.yaml, roles and roles_mapping files. In case the config is not updated as expected.

manoj · May 11, 2022, 2:06am

@Anthony ,
Using admin certificates in securityadmin.sh is throwing “certificate path exception”.

02:03:56.597 [opensearch[_client_][transport_worker][T#1]] ERROR org.opensearch.security.ssl.transport.SecuritySSLNettyTransport - Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

But I’m using the same cert, key pair for connecting 9200 when rest TLS is enabled.
There it’s working fine.
And the CN of the certificate is also mentioned as admin_dn in opensearch.yml

Anthony · May 11, 2022, 8:24am

@manoj You should never use admin certificate for anything else other than making changes that require admin privilege.

In fact there should have been an error when you tried to start the cluster with admin certificates used as node certificates.

I would strongly recommend the following:

Create cluster with separate certificates for admin and node (same node certificates can be used for both transport and http).
Run the securityadmin.sh script to retrieve the config.
Compare it to desired config.
Run below command to examine the roles mapped to anonymous:

curl -XGET "http://localhost:9200/_opendistro/_security/authinfo?pretty"

Please paste the result here

manoj · May 11, 2022, 11:34am

@Anthony ,

But I’m using the same cert, key pair for connecting 9200 when rest TLS is enabled.

Here, I meant I’m using the same client certificate to connect to https://localhost:9200 with TLS enabled.
I’ve three certificates.

Internode certificate for transport (same certificate act as server and client) (configured in opensearch.yml)
HTTP Server (configured in opensearch.yml)
HTTP Client (which is admin certificate) (can be used to connect to localhost:9200 as client)

In working node (deployed multiple times to get it worked 1 time)
curl -XGET "http://localhost:9200/_opendistro/_security/authinfo?pretty" { "user" : "User [name=opendistro_security_anonymous, backend_roles=[opendistro_security_anonymous_backendrole], requestedTenant=null]", "user_name" : "opendistro_security_anonymous", "user_requested_tenant" : null, "remote_address" : "127.0.0.1:46722", "backend_roles" : [ "opendistro_security_anonymous_backendrole" ], "custom_attribute_names" : [ ], "roles" : [ "own_index", "anonymous" ], "tenants" : { "opendistro_security_anonymous" : true }, "principal" : null, "peer_certificates" : "0", "sso_logout_url" : null }

In non-working node,
curl -XGET "http://localhost:9200/_opendistro/_security/authinfo?pretty" Authentication finally failed

Anthony · May 11, 2022, 11:54am

@manoj How many nodes do you have running? Are they all master and data or is there a separation.
Is the opensearch.yaml config the same on all of them?

The below is certainly incorrect:

plugins.security.nodes_dn:
  - CN=localhost
plugins.security.authcz.admin_dn:
  - CN=localhost

This might indeed be a certificate issue if the cluster is not forming properly, do you see any errors in the logs during cluster formation?
To get this working the quickest, I would recommend to follow this to create certificates, get a cluster up and running using these certificates and switching on the basic_auth.

Once this is working as expected, you can enable the anonymous mode and switch off the basic_auth.
Otherwise there might be too many parts causing issues to debug this properly.

manoj · May 12, 2022, 4:27am

@Anthony ,
We have 2 data nodes, 3 master nodes, 2 ingest nodes (1 with rest TLS enabled, 1 with rest TLS disabled) in kubernetes environment.

Here, ingest node with rest TLS enabled is working fine (we can able to do curl -u admin:admin https://localhost:9200 and curl <pass admin/client certificate as options> https://localhost:9200).

We’re trying to achieve curl localhost:9200 in ingest node with rest TLS disabled.

plugins.security.nodes_dn:
  - CN=localhost
plugins.security.authcz.admin_dn:
  - CN=localhost

We created both transport and client (or admin) certificates with CN as localhost.
Will this cause any issue ?
Or do we have command to verify the certificates ?

Sometimes, this (curl localhost:9200) is working fine in rest TLS disabled ingest pod.

Anthony · May 12, 2022, 9:07am

@manoj
I would suggest to follow the previous example. Create certificates as per the docs, leave the TLS on http off (plugins.security.ssl.http.enabled: false) and have the correct (different) certs for nodes and admin (this is a definite issue regardless).

Once everything works using basic_auth, switch it off and switch on anonymous.

Topic		Replies	Views
OpenSearch Keycloak (OpenID) authentication finally failed Security troubleshoot , configure	1	335	February 26, 2024
Transport Client Authentication Error Security troubleshoot , configure	41	132	September 2, 2024
Proxy_auth login failure Security	6	679	December 16, 2021
Disable Rest Layer SSL with Security Plugin Enabled Security troubleshoot , configure	5	93	July 17, 2024
No permissions for user Security	37	6139	July 12, 2023

Authentication finally failed for anonymous user over http randomly

Related Topics