Securityadmin.sh returning permission error for [cluster:monitor/health]

Hi guys

I am trying to setup saml on kibana and attempted to update security via config.yml with command:

sh securityadmin.sh -cd ../securityconfig/ -h xxxx.xxx.xxx.xxx -p 9301 -icl -noopenssl -nhnv -cacert /usr/share/elasticsearch/config/rootCA.pem -cert /usr/share/elasticsearch/config/admin.pem -key /usr/share/elasticsearch/config/admin-key.pem -t config

I am getting the following result

Connected as EMAILADDRESS=xxxxx,CN= xxxxx.xxx.xxx,OU= xxxxx,O= xxxxx,L=xxxxx,C= xxxxx
Elasticsearch Version: 7.10.2
Open Distro Security Version: 1.13.0.0
Contacting elasticsearch cluster 'xxxxxx' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: no permissions for [cluster:monitor/health] and User [name= EMAILADDRESS=xxxxx,CN= xxxxx.xxx.xxx,OU= xxxxx,O= xxxxx,L=xxxxx,C= xxxxx, backend_roles=[], requestedTenant=null]. This is not an error, will keep on trying ...
  Root cause: ElasticsearchSecurityException[no permissions for [cluster:monitor/health] and User [name= EMAILADDRESS=xxxxx,CN= xxxxx.xxx.xxx,OU= xxxxx,O= xxxxx,L=xxxxx,C= xxxxx, backend_roles=[], requestedTenant=null]] (org.elasticsearch.ElasticsearchSecurityException/org.elasticsearch.ElasticsearchSecurityException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

admin_dn has been set in elasticsearch yml:

opendistro_security.ssl.http.pemtrustedcas_filepath: rootCA.pem
opendistro_security.allow_unsafe_democertificates: false
opendistro_security.allow_default_init_securityindex: true
opendistro_security.authcz.admin_dn:
    - “EMAILADDRESS=xxxxx,CN= xxxxx.xxx.xxx,OU= xxxxx,O= xxxxx,L=xxxxx,C= xxxxx”

Other things to point out is that when I log into kibana as admin (changed password from default), I do not see the security plugin on the menu dropdown.

I believe this is an authorization issue but not sure how to proceed. Would greatly appreciate some help on this.

Thanks in advance.

Hi @rjr, couple of questions.
Do you get a response from elasticsearch when you run the below command (not sure if you have http or https):
curl -XGET http://localhost:9200 -u ‘admin:{password}’ --insecure

How did you generate the certificates? Does the DN of the admin cert match the above exactly?
Did you restart the elasticsearch after adding opendistro_security.authcz.admin_dn?

Does running the command with --diagnose provide further details? (Don’t thing -t config option is necessary in any case)

Hello @Anthony

We are running https and can get valid response with:

curl -XGET https://{ hostname or host IP }:9201/ -u ‘admin:{password}’ --insecure
Note. We’ve changed the ports to accomodate two clusters (one ES) on the same host.

They are Internal PKI certs and yes, the admin_dn matches and have restarted.

–diagnose still only displays the permissions error but with more info on which part of the code it trips up.
-t config omitted with no luck.

@rjr
How did you create the admin certs? Have you compared it to the demo admin cert generated via vanilla odfe image?

Regarding the admin user not being able to see security section in menu, this is very strange, as admin role comes with backend role “admin”, which by default maps to “all_access” role, which by default provides access to the security configuration via below entry in elasticsearch.yml file:
opendistro_security.restapi.roles_enabled: ["all_access"]

In order to troubleshoot, run the below api call:

curl --insecure -u admin:{password} -XGET "https://hostname:9201/_opendistro/_security/authinfo?pretty"

make sure under backend_roles, you see “admin”
and under roles you have “all_access”.

and elasticsearch,yml has the entry listed above.

We issued admin and node certs from our internal root CA. The only difference I can see from the certs is that we’ve added email.

As shown in the response below, I believe the admin role configuration is all there

{
“user” : “User [name=admin, backend_roles=[admin], requestedTenant=null]”,
“user_name” : “admin”,
“user_requested_tenant” : null,
“remote_address” : “xxx.xxx.xxx.xxx:xxxxx”,
“backend_roles” : [
“admin”
],
“custom_attribute_names” : ,
“roles” : [
“own_index”,
“all_access”
],
“tenants” : {
“global_tenant” : true,
“admin_tenant” : true,
“admin” : true
},
“principal” : null,
“peer_certificates” : “0”,
“sso_logout_url” : null
}

Anything here that doesn’t look right?

@rjr the user/role looks good, can you provide the elasticsearch.yml file?
Also, are you sure there are no spaces in the admin cert DN?

@Anthony

Please view the elasticsearch.yml below

cluster.name: <cluster name>

discovery.seed_hosts:
<hosts>

cluster.initial_master_nodes:
<nodes>

bootstrap.memory_lock: true # along with the memlock settings below, disables swapping
http.compression: true
transport.tcp.compress: true
transport.port: 9301
http.port: 9201
transport.profiles.default.port: 9301

node.name: xxx.xxx.xxx.xxx
network.host: <node.name>
network.bind_host: <node.ip>
network.publish_host: <node.name>
transport.host: <node.name>
transport.bind_host: <node.ip>
transport.publish_host: <node.name>

cluster.routing.allocation.disk.watermark.low: 92%
cluster.routing.allocation.disk.watermark.high: 95%
cluster.routing.allocation.disk.watermark.flood_stage: 96%

opendistro_security.ssl.transport.pemcert_filepath: node.pem
opendistro_security.ssl.transport.pemkey_filepath: node-key.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: root.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: node.pem
opendistro_security.ssl.http.pemkey_filepath: node-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: root.pem
opendistro_security.allow_unsafe_democertificates: false
opendistro_security.allow_default_init_securityindex: true
opendistro_security.authcz.admin_dn:
  - "EMAILADDRESS=<email>,CN=admin.<domain>,OU=<unit name>,O=<company name>,L=<city>,C=<country>"
opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

The only spaces I see are within the OU and O to separate the names. i.e. “,O=Open Search,”

and my cert dn reads

openssl x509 -noout -in admin.pem -subject | sed ‘s/subject= //’

/C=<country>/L=<city>/O=<company name>/OU=<unit name>/CN=admin.<domain>/emailAddress=<email>

The spaces used are the same between the cert dn and elasticsearch.yml dn

@rjr
Regarding the line in elasticsearch file for admin_dn, you can run the below command to retrieve the correct string that needs to be entered, this is taken from the certificate docs here

openssl x509 -subject -nameopt RFC2253 -noout -in node.pem

Can you confirm that cluster forms properly and you are able to run below command to retrieve the information on the nodes making up the cluster?

Regarding not being able to see the security config option in the UI using admin account, this is very strange indeed, because the role all_access is being mapped (as you confirmed with _opendistro/_security/authinfo api).

I would recommend to spin up a local single node cluster and remove the extra configs from elasticsearch.yml, to make sure everything works as expected. There maybe some confusion regarding the multiple clusters on a single host.

@Anthony cheers for the cert query and documentation. It seems there is a slight difference at the start of the dn string.

The query results with “subject= emailAddress=…” Does casing matter?

In elasticsearch.yml my admin_dn starts with “EMAILADDRESS=…” instead. I tried replacing with “emailAddress=” which produced the same error. Also tried “subject= emailAddress=…” but that gave me error

ERR: EMAILADDRESS=<email>,CN=admin.<domain>,OU=<unit name>,O=<company name>,L=<city>,C=<country> is not an admin user

Just wondering, did you want me to run another command but forgot to add it in your last message?

I’ll let you know if I end up spinning up the test instance.

Now that I think about it… to change the admin password, I believe I needed to run the securityadmin.sh to apply that change so i’m really not sure why it’s not accepting the admin cert anymore.

@rjr
the casing doesn’t make any difference and it’s only the part after "subject= " that needs to be listed.

Regarding the missing command, yes, I was meant to include:

curl -XGET https://localhost:9200/_cat/nodes?pretty -u 'admin:admin' --insecure

During your local testing, can you remove email from subject in certificates, just to eliminate possible issues.

Hey @Anthony

Got it working now! It was due to this line

opendistro_security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]

being missing in some of my ES nodes. I am able to successfully run the security admin script and can see the security module in Kibana.

Cheers for the help earlier.