# Open Search Security Not Initialized

**URL:** <https://forum.opensearch.org/t/open-search-security-not-initialized/10537>\
**Category:** Security\
**Tags:** troubleshoot\
**Created:** [August 8, 2022, 4:31pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537 "2022-08-08T16:31:36Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ddodoo](https://avatars.discourse-cdn.com/v4/letter/d/bcef8e/32.png) [@ddodoo](https://forum.opensearch.org/u/ddodoo)\
**Post date:** [August 8, 2022, 4:31pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/1 "2022-08-08T16:31:36Z")

</div>

I run into an error when excecuting the securityadmin.sh script from the [Documentation](https://opensearch.org/docs/latest/security-plugin/configuration/security-admin/#a-word-of-caution)

./securityadmin.sh -cd …/…/…/config/opensearch-security/ -icl -nhnv  
-cacert …/…/…/config/root-ca.pem  
-cert …/…/…/config/kirk.pem  
-key …/…/…/config/kirk-key.pem

**I receive the following error response** :

```nohighlight
Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates) 
Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml 
If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

```

Can you help me troubleshoot the possible cause?

Kindly find my log file attached:  
[securityadmin\_diag\_trace](https://github.com/opensearch-project/helm-charts/files/9281029/securityadmin_diag_trace_2022-Aug-08_09-17-28.txt)

---

<div class="post-metadata">

**Author:** ![nateynate](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/nateynate/32/4913_2.png) [@nateynate](https://forum.opensearch.org/u/nateynate)\
**Post date:** [August 8, 2022, 5:26pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/2 "2022-08-08T17:26:25Z")

</div>

Hi @ddodoo -

I’m noticing two things here that I’d like to verify before I jump to conclusions. First, it looks like you’re invoking `securityadmin.sh` but passing references to files using a series of three dots instead of two. Is this intentional, or just a result of pasting the output into the forum here?

> [@ddodoo](#):
>
> ./securityadmin.sh -cd …/…/…/config/opensearch-security/ -icl -nhnv  
> -cacert …/…/…/config/root-ca.pem  
> -cert …/…/…/config/kirk.pem  
> -key …/…/…/config/kirk-key.pem

Secondly, I’m seeing this in the trace:

```auto
OpenSearchStatusException[OpenSearch exception [type=master_not_discovered_exception, reason=null]]
	at org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:207)
	at org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2081)
	at org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2058)
	at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1777)
	at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1730)
	at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1698)
	at org.opensearch.client.ClusterClient.health(ClusterClient.java:162)
	at org.opensearch.security.tools.SecurityAdmin.generateDiagnoseTrace(SecurityAdmin.java:999)
	at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:595)
	at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:159)
	Suppressed: org.opensearch.client.ResponseException: method [GET], host [https://localhost:9200], URI [/_cluster/health?master_timeout=30s&level=cluster&timeout=30s], status line [HTTP/1.1 503 Service Unavailable]
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
		at org.opensearch.client.RestClient.convertResponse(RestClient.java:350)
		at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
		at org.opensearch.client.RestClient.performRequest(RestClient.java:295)
		at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764)
		... 6 more

```

I’m a bit out of my element here, but let’s try without the triple dots. 🙂

Nate

---

<div class="post-metadata">

**Author:** ![ddodoo](https://avatars.discourse-cdn.com/v4/letter/d/bcef8e/32.png) [@ddodoo](https://forum.opensearch.org/u/ddodoo)\
**Post date:** [August 8, 2022, 5:43pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/3 "2022-08-08T17:43:16Z")

</div>

> [@nateynate](#):
>
> I’m noticing two things here that I’d like to verify before I jump to conclusions. First, it looks like you’re invoking `securityadmin.sh` but passing references to files using a series of three dots instead of two. Is this intentional, or just a result of pasting the output into the forum here?

Hello @nateynate , I examined both the original documentation and the script I executed in the terminal

It has 2 dots and not 3

I guess the 3 dots is from the forum page formatting

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [August 8, 2022, 6:08pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/4 "2022-08-08T18:08:12Z")

</div>

@ddodoo According to the logs, service at port 9200 is not up.  
Do you see any other errors? Did your nodes initialize?

Please share your config.yml and opensearch.yml files.  
How many nodes do you have in the cluster?

Also, please share the output of the below command.

```auto
curl --insecure -u admin:admin -XGET https://localhost:9200/

```

---

<div class="post-metadata">

**Author:** ![ddodoo](https://avatars.discourse-cdn.com/v4/letter/d/bcef8e/32.png) [@ddodoo](https://forum.opensearch.org/u/ddodoo)\
**Post date:** [August 9, 2022, 7:12am UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/5 "2022-08-09T07:12:24Z")

</div>

I followed the port-forward command from the OpenSearch Multi-Node Cluster Kubernetes documentation @pablo

[Setup OpenSearch multi-node cluster on Kubernetes using Helm Charts · OpenSearch](https://opensearch.org/blog/technical-posts/2021/11/setup-multinode-cluster-kubernetes/)

```auto
 kubectl port-forward opensearch-cluster-master-0 9200

```

I have a single node

The output of the `curl --insecure -u admin:admin -XGET https://localhost:9200/`  
is **“OpenSearch Security not initialized”**

Kindly find the image attached :

 ![Screenshot (377)](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/2/2092253d0c5f96d02e3fa802a7509e7f5e1e3ca1.png)

Kindly find the **config.yml** and **opensearch.yml** files below:

**config.yml**

```auto
---

# This is the main OpenSearch Security configuration file where authentication
# and authorization is defined.
#
# You need to configure at least one authentication domain in the authc of this file.
# An authentication domain is responsible for extracting the user credentials from
# the request and for validating them against an authentication backend like Active Directory for example.
#
# If more than one authentication domain is configured the first one which succeeds wins.
# If all authentication domains fail then the request is unauthenticated.
# In this case an exception is thrown and/or the HTTP status is set to 401.
#
# After authentication authorization (authz) will be applied. There can be zero or more authorizers which collect
# the roles from a given backend for the authenticated user.
#
# Both, authc and auth can be enabled/disabled separately for REST and TRANSPORT layer. Default is true for both.
# http_enabled: true
# transport_enabled: true
#
# For HTTP it is possible to allow anonymous authentication. If that is the case then the HTTP authenticators try to
# find user credentials in the HTTP request. If credentials are found then the user gets regularly authenticated.
# If none can be found the user will be authenticated as an "anonymous" user. This user has always the username "anonymous"
# and one role named "anonymous_backendrole".
# If you enable anonymous authentication all HTTP authenticators will not challenge.
#
#
# Note: If you define more than one HTTP authenticators make sure to put non-challenging authenticators like "proxy" or "clientcert"
# first and the challenging one last.
# Because it's not possible to challenge a client with two different authentication methods (for example
# Kerberos and Basic) only one can have the challenge flag set to true. You can cope with this situation
# by using pre-authentication, e.g. sending a HTTP Basic authentication header in the request.
#
# Default value of the challenge flag is true.
#
#
# HTTP
# basic (challenging)
# proxy (not challenging, needs xff)
# kerberos (challenging)
# clientcert (not challenging, needs https)
# jwt (not challenging)
# host (not challenging) #DEPRECATED, will be removed in a future version.
# host based authentication is configurable in roles_mapping

# Authc
# internal
# noop
# ldap

# Authz
# ldap
# noop

_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
    # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
    # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
    #filtered_alias_mode: warn
    #do_not_fail_on_forbidden: false
    #kibana:
    # Kibana multitenancy
    #multitenancy_enabled: true
    #server_username: kibanaserver
    #index: '.kibana'
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
        #internalProxies: '.*' # trust all internal proxies, regex pattern
        #remoteIpHeader: 'x-forwarded-for'
        ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
        ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
        ###### and here https://tools.ietf.org/html/rfc7239
        ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
    authc:
      kerberos_auth_domain:
        http_enabled: false
        transport_enabled: false
        order: 6
        http_authenticator:
          type: kerberos
          challenge: true
          config:
            # If true a lot of kerberos/security related debugging output will be logged to standard out
            krb_debug: false
            # If true then the realm will be stripped from the user name
            strip_realm_from_principal: true
        authentication_backend:
          type: noop
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 4
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
      proxy_auth_domain:
        description: "Authenticate via proxy"
        http_enabled: false
        transport_enabled: false
        order: 3
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop
      jwt_auth_domain:
        description: "Authenticate via Json Web Token"
        http_enabled: false
        transport_enabled: false
        order: 0
        http_authenticator:
          type: jwt
          challenge: false
          config:
            signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
            jwt_header: "Authorization"
            jwt_url_parameter: null
            roles_key: null
            subject_key: null
        authentication_backend:
          type: noop
      clientcert_auth_domain:
        description: "Authenticate via SSL client certificates"
        http_enabled: false
        transport_enabled: false
        order: 2
        http_authenticator:
          type: clientcert
          config:
            username_attribute: cn #optional, if omitted DN becomes username
          challenge: false
        authentication_backend:
          type: noop
      ldap:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        order: 5
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          # LDAP authentication backend (authenticate users against a LDAP or Active Directory)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - localhost:8389
            bind_dn: null
            password: null
            userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(sAMAccountName={0})'
            # Use this attribute from the user as username (if not set then DN is used)
            username_attribute: null
    authz:
      roles_from_myldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - localhost:8389
            bind_dn: null
            password: null
            rolebase: 'ou=groups,dc=example,dc=com'
            # Filter to search for roles (currently in the whole subtree beneath rolebase)
            # {0} is substituted with the DN of the user
            # {1} is substituted with the username
            # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
            rolesearch: '(member={0})'
            # Specify the name of the attribute which value should be substituted with {2} above
            userroleattribute: null
            # Roles as an attribute of the user entry
            userrolename: disabled
            #userrolename: memberOf
            # The attribute in a role entry containing the name of that role, Default is "name".
            # Can also be "dn" to use the full DN as rolename.
            rolename: cn
            # Resolve nested roles transitive (roles which are members of other roles and so on ...)
            resolve_nested_roles: true
            userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(uid={0})'
            # Skip users matching a user name, a wildcard or a regex pattern
            #skip_users:
            # - 'cn=Michael Jackson,ou*people,o=TEST'
            # - '/\S*/'
      roles_from_another_ldap:
        description: "Authorize via another Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: ldap
          #config goes here ...
  # auth_failure_listeners:
  # ip_rate_limiting:
  # type: ip
  # allowed_tries: 10
  # time_window_seconds: 3600
  # block_expiry_seconds: 600
  # max_blocked_clients: 100000
  # max_tracked_clients: 100000
  # internal_authentication_backend_limiting:
  # type: username
  # authentication_backend: intern
  # allowed_tries: 10
  # time_window_seconds: 3600
  # block_expiry_seconds: 600
  # max_blocked_clients: 100000
  # max_tracked_clients: 100000

```

**opensearch.yml**

```auto
cluster.name: opensearch-cluster

# Bind to all interfaces because we don't know what IP address Docker will assign to us.
network.host: 0.0.0.0

# Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
# discovery.type: single-node

# Start OpenSearch Security Demo Configuration
# WARNING: revise all the lines below before you go into production
plugins:
  security:
    ssl:
      transport:
        pemcert_filepath: esnode.pem
        pemkey_filepath: esnode-key.pem
        pemtrustedcas_filepath: root-ca.pem
        enforce_hostname_verification: false
      http:
        enabled: true
        pemcert_filepath: esnode.pem
        pemkey_filepath: esnode-key.pem
        pemtrustedcas_filepath: root-ca.pem
    allow_unsafe_democertificates: true
    allow_default_init_securityindex: true
    authcz:
      admin_dn:
        - CN=kirk,OU=client,O=client,L=test,C=de
    audit.type: internal_opensearch
    enable_snapshot_restore_privilege: true
    check_snapshot_restore_write_privileges: true
    restapi:
      roles_enabled: ["all_access", "security_rest_api_access"]
    system_indices:
      enabled: true
      indices:
        [
          ".opendistro-alerting-config",
          ".opendistro-alerting-alert*",
          ".opendistro-anomaly-results*",
          ".opendistro-anomaly-detector*",
          ".opendistro-anomaly-checkpoints",
          ".opendistro-anomaly-detection-state",
          ".opendistro-reports-*",
          ".opendistro-notifications-*",
          ".opendistro-notebooks",
          ".opendistro-asynchronous-search-response*",
        ]
######## End OpenSearch Security Demo Configuration ########

```

---

<div class="post-metadata">

**Author:** ![ddodoo](https://avatars.discourse-cdn.com/v4/letter/d/bcef8e/32.png) [@ddodoo](https://forum.opensearch.org/u/ddodoo)\
**Post date:** [August 9, 2022, 7:14am UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/6 "2022-08-09T07:14:55Z")

</div>

**Port Forwarding**

 ![Screenshot (378)](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/6/64eccb45b89ac9635c706bf8c9a7c89052284665.png)

@pablo

---

<div class="post-metadata">

**Author:** ![ddodoo](https://avatars.discourse-cdn.com/v4/letter/d/bcef8e/32.png) [@ddodoo](https://forum.opensearch.org/u/ddodoo)\
**Post date:** [August 9, 2022, 7:15am UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/7 "2022-08-09T07:15:47Z")

</div>

**Executing securityadmin.sh Script**

 ![Screenshot (379)](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/9/9a9c0e007badc57ebe8f57daf89a267f6c4fa885.png)

@pablo

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [August 9, 2022, 1:03pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/8 "2022-08-09T13:03:58Z")

</div>

@ddodoo @ddodoo I understand that you’ve followed `Setup OpenSearch multi-node cluster on Kubernetes using Helm Charts` and you have created a single-node cluster and not multi-node as described in the setup.

According to your screenshots, you run opensearch-cluster-master-0 pod. Did you follow master.yaml config and set the below values?

 ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/a/ab66767ffeb011c2815d12ab9411188f89db39ac.png)

If so, then your cluster doesn’t have any node with `data` role. Could you confirm that?  
Also, did you set `discovery.type:` to `single-node`

---

<div class="post-metadata">

**Author:** ![ddodoo](https://avatars.discourse-cdn.com/v4/letter/d/bcef8e/32.png) [@ddodoo](https://forum.opensearch.org/u/ddodoo)\
**Post date:** [August 10, 2022, 2:02pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/9 "2022-08-10T14:02:04Z")

</div>

Yes I have the same configuration to my master.yaml file @pablo

How do I set `discovery.type:` to `single-node`?

```auto
# These will be set as environment variable "node. Roles". E.g. node.roles=master,ingest,data,remote_cluster_client
roles:
  - master: "true"
  - ingest: "false"
  - data: "false"
  - remote_cluster_client: "false"

replicas: 1

```

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [August 10, 2022, 2:28pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/10 "2022-08-10T14:28:36Z")

</div>

@ddodoo The main problem is in the roles. If you’ve just used master.yaml then that master node has no data role. Without a data role, no indices, including .opendistro\_security, will be created.  
You should be already seeing errors in the OpenSearch pod’s logs regarding the security configuration.

The example described in the manual creates 3 nodes cluster.

The quick fix is to set all the flags to `true` on your single master node.

This should work without setting `single-node`. If not, try setting it in the opensearch.yml section in master.yaml file.

---

<div class="post-metadata">

**Author:** ![ddodoo](https://avatars.discourse-cdn.com/v4/letter/d/bcef8e/32.png) [@ddodoo](https://forum.opensearch.org/u/ddodoo)\
**Post date:** [August 10, 2022, 4:32pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/11 "2022-08-10T16:32:15Z")

</div>

I used helm to install all 3 yaml files (master.yaml, client.yaml and data.yaml) as suggested in the documentation and not just the master.yaml file

@pablo

---

<div class="post-metadata">

**Author:** ![pauchamb](https://avatars.discourse-cdn.com/v4/letter/p/7ea924/32.png) [@pauchamb](https://forum.opensearch.org/u/pauchamb)\
**Post date:** [August 10, 2022, 4:40pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/12 "2022-08-10T16:40:04Z")

</div>

Hey @ddodoo

I ran into a similar issue like this. The difference between our environments is that I installed OpenSearch via an RPM but the two settings I needed when first starting up my cluster were:

```auto
plugins.security.allow_default_init_securityindex: true
cluster.initial_cluster_manager_nodes: [list, of, nodes]

```

Once I had those in opensearch.yml and restarted, the cluster initialized just fine with any number of nodes.

---

<div class="post-metadata">

**Author:** ![ddodoo](https://avatars.discourse-cdn.com/v4/letter/d/bcef8e/32.png) [@ddodoo](https://forum.opensearch.org/u/ddodoo)\
**Post date:** [August 10, 2022, 6:29pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/13 "2022-08-10T18:29:38Z")

</div>

Is this already in the opensearch.yml file or I have to add it? @pauchamb

```auto
plugins.security.allow_default_init_securityindex: true
cluster.initial_cluster_manager_nodes: [list, of, nodes]

```

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [August 11, 2022, 10:38am UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/14 "2022-08-11T10:38:15Z")

</div>

@ddodoo Could you share the output of the below command?

```auto
kubectl get all -A

```

---

<div class="post-metadata">

**Author:** ![ddodoo](https://avatars.discourse-cdn.com/v4/letter/d/bcef8e/32.png) [@ddodoo](https://forum.opensearch.org/u/ddodoo)\
**Post date:** [August 11, 2022, 9:59pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/15 "2022-08-11T21:59:33Z")

</div>

> [@pablo](#):
>
> ```auto
> kubectl get all -A
> 
> ```

Sure sure

Kindly view the response @pablo

```auto
root@ubuntu-test-vm:/home/azureuser# kubectl get all -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/dashboards-opensearch-dashboards-584c75fcb-zfqld 1/1 Running 0 21m
default pod/opensearch-cluster-client-0 1/1 Running 0 31m
default pod/opensearch-cluster-data-0 1/1 Running 0 32m
default pod/opensearch-cluster-master-0 1/1 Running 0 32m
kube-system pod/coredns-6d4b75cb6d-p9zwq 1/1 Running 0 37m
kube-system pod/etcd-minikube 1/1 Running 0 37m
kube-system pod/kube-apiserver-minikube 1/1 Running 0 37m
kube-system pod/kube-controller-manager-minikube 1/1 Running 0 37m
kube-system pod/kube-proxy-vjstv 1/1 Running 0 37m
kube-system pod/kube-scheduler-minikube 1/1 Running 0 37m
kube-system pod/storage-provisioner 1/1 Running 1 (36m ago) 37m

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/dashboards-opensearch-dashboards ClusterIP 10.96.13.93 <none> 5601/TCP 21m
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 37m
default service/opensearch-cluster-client ClusterIP 10.106.244.135 <none> 9200/TCP,9300/TCP 31m
default service/opensearch-cluster-client-headless ClusterIP None <none> 9200/TCP,9300/TCP 31m
default service/opensearch-cluster-data ClusterIP 10.109.206.142 <none> 9200/TCP,9300/TCP 32m
default service/opensearch-cluster-data-headless ClusterIP None <none> 9200/TCP,9300/TCP 32m
default service/opensearch-cluster-master ClusterIP 10.99.170.57 <none> 9200/TCP,9300/TCP 32m
default service/opensearch-cluster-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 32m
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 37m

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 37m

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
default deployment.apps/dashboards-opensearch-dashboards 1/1 1 1 21m
kube-system deployment.apps/coredns 1/1 1 1 37m

NAMESPACE NAME DESIRED CURRENT READY AGE
default replicaset.apps/dashboards-opensearch-dashboards-584c75fcb 1 1 1 21m
kube-system replicaset.apps/coredns-6d4b75cb6d 1 1 1 37m

NAMESPACE NAME READY AGE
default statefulset.apps/opensearch-cluster-client 1/1 31m
default statefulset.apps/opensearch-cluster-data 1/1 32m
default statefulset.apps/opensearch-cluster-master 1/1 32m

```

---

<div class="post-metadata">

**Author:** ![ddodoo](https://avatars.discourse-cdn.com/v4/letter/d/bcef8e/32.png) [@ddodoo](https://forum.opensearch.org/u/ddodoo)\
**Post date:** [August 11, 2022, 10:01pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/16 "2022-08-11T22:01:28Z")

</div>

I cannot make changes to the pod container because the opensearch.yml file is a read-only file?

How do I resolve this?

@pauchamb

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [August 12, 2022, 10:27am UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/17 "2022-08-12T10:27:38Z")

</div>

@ddodoo Could you run the following commands and share the files and outputs?

1. Logs from each OpenSearch container

```auto
kubectl logs opensearch-cluster-client-0 > opensearch-cluster-client-0.log
kubectl logs opensearch-cluster-master-0 > opensearch-cluster-master-0.log
kubectl logs opensearch-cluster-data-0 > opensearch-cluster-data-0.log

```

1. Run port-forwarding for data node and run curl against that.

```auto
kubectl port-forward opensearch-cluster-data-0 9200

curl --insecure -u admin:admin -XGET https://localhost:9200/

```

---

<div class="post-metadata">

**Author:** ![ddodoo](https://avatars.discourse-cdn.com/v4/letter/d/bcef8e/32.png) [@ddodoo](https://forum.opensearch.org/u/ddodoo)\
**Post date:** [August 12, 2022, 10:40pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/18 "2022-08-12T22:40:00Z")

</div>

@pablo

**opensearch-cluster-master-0**

```auto
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
**************************************************************************
 **This tool will be deprecated in the next major release of OpenSearch**
 **https://github.com/opensearch-project/security/issues/1755**
 **************************************************************************
OpenSearch Security Demo Installer
 **Warning: Do not use on production or public reachable systems**
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-2.2.0
Detected OpenSearch Security Version: 2.2.0.0

tee: /usr/share/opensearch/config/opensearch.yml: Permission denied
Enabling OpenSearch Security Plugin
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.2.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.2.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2022-08-12T17:18:38,737][INFO][o.o.n.Node] [opensearch-cluster-master-0] version[2.2.0], pid[45], build[tar/b1017fa3b9a1c781d4f34ecee411e0cdf930a515/2022-08-09T02:27:25.256769336Z], OS[Linux/5.4.0-1089-azure/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.4/17.0.4+8]
[2022-08-12T17:18:38,739][INFO][o.o.n.Node] [opensearch-cluster-master-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[2022-08-12T17:18:38,739][INFO][o.o.n.Node] [opensearch-cluster-master-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-5998433139062988299, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx512M, -Xms512M, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2022-08-12T17:18:40,058][WARN][stderr] [opensearch-cluster-master-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2022-08-12T17:18:40,059][WARN][stderr] [opensearch-cluster-master-0] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2022-08-12T17:18:40,059][WARN][stderr] [opensearch-cluster-master-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2022-08-12T17:18:40,071][INFO][o.o.s.s.t.SSLConfig] [opensearch-cluster-master-0] SSL dual mode is disabled
[2022-08-12T17:18:40,071][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] OpenSearch Config path is /usr/share/opensearch/config
[2022-08-12T17:18:40,470][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] JVM supports TLSv1.3
[2022-08-12T17:18:40,472][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2022-08-12T17:18:40,745][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] TLS Transport Client Provider : JDK
[2022-08-12T17:18:40,746][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] TLS Transport Server Provider : JDK
[2022-08-12T17:18:40,746][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] TLS HTTP Provider : JDK
[2022-08-12T17:18:40,746][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2022-08-12T17:18:40,746][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[2022-08-12T17:18:41,113][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] Clustername: opensearch-cluster
[2022-08-12T17:18:41,118][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[2022-08-12T17:18:41,118][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[2022-08-12T17:18:41,119][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/kirk.pem has insecure file permissions (should be 0600)
[2022-08-12T17:18:41,119][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
[2022-08-12T17:18:41,119][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/esnode-key.pem has insecure file permissions (should be 0600)
[2022-08-12T17:18:41,119][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600)
[2022-08-12T17:18:41,119][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/kirk-key.pem has insecure file permissions (should be 0600)
[2022-08-12T17:18:41,938][INFO][o.o.p.c.PluginSettings] [opensearch-cluster-master-0] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2022-08-12T17:18:42,639][INFO][o.o.i.r.ReindexPlugin] [opensearch-cluster-master-0] ReindexPlugin reloadSPI called
[2022-08-12T17:18:42,640][INFO][o.o.i.r.ReindexPlugin] [opensearch-cluster-master-0] Unable to find any implementation for RemoteReindexExtension
[2022-08-12T17:18:42,714][INFO][o.o.j.JobSchedulerPlugin] [opensearch-cluster-master-0] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2022-08-12T17:18:42,741][INFO][o.o.j.JobSchedulerPlugin] [opensearch-cluster-master-0] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2022-08-12T17:18:42,742][INFO][o.o.j.JobSchedulerPlugin] [opensearch-cluster-master-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-08-12T17:18:42,748][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [aggs-matrix-stats]
[2022-08-12T17:18:42,748][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [analysis-common]
[2022-08-12T17:18:42,748][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [geo]
[2022-08-12T17:18:42,748][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [ingest-common]
[2022-08-12T17:18:42,748][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [ingest-geoip]
[2022-08-12T17:18:42,748][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [ingest-user-agent]
[2022-08-12T17:18:42,748][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [lang-expression]
[2022-08-12T17:18:42,748][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [lang-mustache]
[2022-08-12T17:18:42,748][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [lang-painless]
[2022-08-12T17:18:42,748][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [mapper-extras]
[2022-08-12T17:18:42,748][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [opensearch-dashboards]
[2022-08-12T17:18:42,748][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [parent-join]
[2022-08-12T17:18:42,748][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [percolator]
[2022-08-12T17:18:42,748][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [rank-eval]
[2022-08-12T17:18:42,749][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [reindex]
[2022-08-12T17:18:42,749][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [repository-url]
[2022-08-12T17:18:42,749][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [systemd]
[2022-08-12T17:18:42,749][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded module [transport-netty4]
[2022-08-12T17:18:42,749][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-alerting]
[2022-08-12T17:18:42,749][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-anomaly-detection]
[2022-08-12T17:18:42,749][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-asynchronous-search]
[2022-08-12T17:18:42,750][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-cross-cluster-replication]
[2022-08-12T17:18:42,750][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-geospatial]
[2022-08-12T17:18:42,750][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-index-management]
[2022-08-12T17:18:42,750][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-job-scheduler]
[2022-08-12T17:18:42,750][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-knn]
[2022-08-12T17:18:42,750][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-ml]
[2022-08-12T17:18:42,750][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-notifications]
[2022-08-12T17:18:42,750][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-notifications-core]
[2022-08-12T17:18:42,750][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-observability]
[2022-08-12T17:18:42,750][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-performance-analyzer]
[2022-08-12T17:18:42,750][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-reports-scheduler]
[2022-08-12T17:18:42,750][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-security]
[2022-08-12T17:18:42,750][INFO][o.o.p.PluginsService] [opensearch-cluster-master-0] loaded plugin [opensearch-sql]
[2022-08-12T17:18:42,768][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2022-08-12T17:18:42,827][INFO][o.o.e.NodeEnvironment] [opensearch-cluster-master-0] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/sda1)]], net usable_space [22.9gb], net total_space [28.8gb], types [ext4]
[2022-08-12T17:18:42,827][INFO][o.o.e.NodeEnvironment] [opensearch-cluster-master-0] heap size [512mb], compressed ordinary object pointers [true]
[2022-08-12T17:18:42,863][INFO][o.o.n.Node] [opensearch-cluster-master-0] node name [opensearch-cluster-master-0], node ID [R2a0BBXMRQ6fpE7RO050dw], cluster name [opensearch-cluster], roles [map[data:false], map[ingest:false], map[master:true], map[remote_cluster_client:false]]
[2022-08-12T17:18:48,253][WARN][o.o.s.c.Salt] [opensearch-cluster-master-0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2022-08-12T17:18:48,326][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-cluster-master-0] Message routing enabled: true
[2022-08-12T17:18:48,364][INFO][o.o.s.f.SecurityFilter] [opensearch-cluster-master-0] <NONE> indices are made immutable.
[2022-08-12T17:18:48,863][INFO][o.o.a.b.ADCircuitBreakerService] [opensearch-cluster-master-0] Registered memory breaker.
[2022-08-12T17:18:49,324][INFO][o.o.m.c.b.MLCircuitBreakerService] [opensearch-cluster-master-0] Registered ML memory breaker.
[2022-08-12T17:18:49,915][INFO][o.o.t.NettyAllocator] [opensearch-cluster-master-0] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
[2022-08-12T17:18:50,012][INFO][o.o.d.DiscoveryModule] [opensearch-cluster-master-0] using discovery type [zen] and seed hosts providers [settings]
[2022-08-12T17:18:50,622][WARN][o.o.g.DanglingIndicesState] [opensearch-cluster-master-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-08-12T17:18:51,313][INFO][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-cluster-master-0] PerformanceAnalyzer Enabled: false
[2022-08-12T17:18:51,342][INFO][o.o.n.Node] [opensearch-cluster-master-0] initialized
[2022-08-12T17:18:51,342][INFO][o.o.n.Node] [opensearch-cluster-master-0] starting ...
[2022-08-12T17:18:51,428][INFO][o.o.t.TransportService] [opensearch-cluster-master-0] publish_address {172.17.0.3:9300}, bound_addresses {0.0.0.0:9300}
[2022-08-12T17:18:51,596][INFO][o.o.b.BootstrapChecks] [opensearch-cluster-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-08-12T17:19:21,612][WARN][o.o.n.Node] [opensearch-cluster-master-0] timed out while waiting for initial discovery state - timeout: 30s
[2022-08-12T17:19:21,619][INFO][o.o.h.AbstractHttpServerTransport] [opensearch-cluster-master-0] publish_address {172.17.0.3:9200}, bound_addresses {0.0.0.0:9200}
[2022-08-12T17:19:21,619][INFO][o.o.n.Node] [opensearch-cluster-master-0] started
[2022-08-12T17:19:21,620][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] Node started
[2022-08-12T17:19:21,620][INFO][o.o.s.c.ConfigurationRepository] [opensearch-cluster-master-0] Will attempt to create index .opendistro_security and default configs if they are absent
[2022-08-12T17:19:21,621][INFO][o.o.s.c.ConfigurationRepository] [opensearch-cluster-master-0] Background init thread started. Install default config?: true
[2022-08-12T17:19:21,621][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] 0 OpenSearch Security modules loaded so far: []
[2022-08-12T17:19:51,628][ERROR][o.o.s.c.ConfigurationRepository] [opensearch-cluster-master-0] Cannot apply default config (this is maybe not an error!)
org.opensearch.discovery.ClusterManagerNotDiscoveredException: null
	at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAction.java:305) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.2.0.jar:2.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T17:19:51,634][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-master-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T17:20:04,638][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-master-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T17:20:17,640][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-master-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]

```

---

<div class="post-metadata">

**Author:** ![ddodoo](https://avatars.discourse-cdn.com/v4/letter/d/bcef8e/32.png) [@ddodoo](https://forum.opensearch.org/u/ddodoo)\
**Post date:** [August 12, 2022, 10:41pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/19 "2022-08-12T22:41:54Z")

</div>

@pablo

**opensearch-cluster-data-0**

```auto
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
**************************************************************************
 **This tool will be deprecated in the next major release of OpenSearch**
 **https://github.com/opensearch-project/security/issues/1755**
 **************************************************************************
OpenSearch Security Demo Installer
 **Warning: Do not use on production or public reachable systems**
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-2.2.0
Detected OpenSearch Security Version: 2.2.0.0
tee: /usr/share/opensearch/config/opensearch.yml: Permission denied

Enabling OpenSearch Security Plugin
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.2.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.2.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2022-08-12T19:12:58,240][INFO][o.o.n.Node] [opensearch-cluster-data-0] version[2.2.0], pid[44], build[tar/b1017fa3b9a1c781d4f34ecee411e0cdf930a515/2022-08-09T02:27:25.256769336Z], OS[Linux/5.4.0-1089-azure/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.4/17.0.4+8]
[2022-08-12T19:12:58,243][INFO][o.o.n.Node] [opensearch-cluster-data-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[2022-08-12T19:12:58,243][INFO][o.o.n.Node] [opensearch-cluster-data-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-3276608203265200185, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx512M, -Xms512M, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2022-08-12T19:12:59,716][WARN][stderr] [opensearch-cluster-data-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2022-08-12T19:12:59,716][WARN][stderr] [opensearch-cluster-data-0] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2022-08-12T19:12:59,717][WARN][stderr] [opensearch-cluster-data-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2022-08-12T19:12:59,732][INFO][o.o.s.s.t.SSLConfig] [opensearch-cluster-data-0] SSL dual mode is disabled
[2022-08-12T19:12:59,732][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] OpenSearch Config path is /usr/share/opensearch/config
[2022-08-12T19:13:00,257][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-data-0] JVM supports TLSv1.3
[2022-08-12T19:13:00,260][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-data-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2022-08-12T19:13:00,944][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-data-0] TLS Transport Client Provider : JDK
[2022-08-12T19:13:00,944][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-data-0] TLS Transport Server Provider : JDK
[2022-08-12T19:13:00,944][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-data-0] TLS HTTP Provider : JDK
[2022-08-12T19:13:00,944][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-data-0] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2022-08-12T19:13:00,944][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-data-0] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[2022-08-12T19:13:01,367][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] Clustername: opensearch-cluster
[2022-08-12T19:13:01,417][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[2022-08-12T19:13:01,418][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[2022-08-12T19:13:01,418][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] File /usr/share/opensearch/config/kirk.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:01,418][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:01,418][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] File /usr/share/opensearch/config/esnode-key.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:01,418][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:01,418][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] File /usr/share/opensearch/config/kirk-key.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:02,134][INFO][o.o.p.c.PluginSettings] [opensearch-cluster-data-0] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2022-08-12T19:13:02,929][INFO][o.o.i.r.ReindexPlugin] [opensearch-cluster-data-0] ReindexPlugin reloadSPI called
[2022-08-12T19:13:02,930][INFO][o.o.i.r.ReindexPlugin] [opensearch-cluster-data-0] Unable to find any implementation for RemoteReindexExtension
[2022-08-12T19:13:03,013][INFO][o.o.j.JobSchedulerPlugin] [opensearch-cluster-data-0] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2022-08-12T19:13:03,037][INFO][o.o.j.JobSchedulerPlugin] [opensearch-cluster-data-0] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2022-08-12T19:13:03,038][INFO][o.o.j.JobSchedulerPlugin] [opensearch-cluster-data-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-08-12T19:13:03,044][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [aggs-matrix-stats]
[2022-08-12T19:13:03,044][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [analysis-common]
[2022-08-12T19:13:03,044][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [geo]
[2022-08-12T19:13:03,044][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [ingest-common]
[2022-08-12T19:13:03,044][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [ingest-geoip]
[2022-08-12T19:13:03,044][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [ingest-user-agent]
[2022-08-12T19:13:03,044][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [lang-expression]
[2022-08-12T19:13:03,044][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [lang-mustache]
[2022-08-12T19:13:03,044][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [lang-painless]
[2022-08-12T19:13:03,044][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [mapper-extras]
[2022-08-12T19:13:03,044][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [opensearch-dashboards]
[2022-08-12T19:13:03,045][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [parent-join]
[2022-08-12T19:13:03,045][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [percolator]
[2022-08-12T19:13:03,045][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [rank-eval]
[2022-08-12T19:13:03,045][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [reindex]
[2022-08-12T19:13:03,045][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [repository-url]
[2022-08-12T19:13:03,045][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [systemd]
[2022-08-12T19:13:03,045][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded module [transport-netty4]
[2022-08-12T19:13:03,045][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-alerting]
[2022-08-12T19:13:03,045][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-anomaly-detection]
[2022-08-12T19:13:03,046][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-asynchronous-search]
[2022-08-12T19:13:03,046][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-cross-cluster-replication]
[2022-08-12T19:13:03,046][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-geospatial]
[2022-08-12T19:13:03,046][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-index-management]
[2022-08-12T19:13:03,046][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-job-scheduler]
[2022-08-12T19:13:03,046][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-knn]
[2022-08-12T19:13:03,046][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-ml]
[2022-08-12T19:13:03,046][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-notifications]
[2022-08-12T19:13:03,046][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-notifications-core]
[2022-08-12T19:13:03,046][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-observability]
[2022-08-12T19:13:03,046][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-performance-analyzer]
[2022-08-12T19:13:03,046][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-reports-scheduler]
[2022-08-12T19:13:03,046][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-security]
[2022-08-12T19:13:03,046][INFO][o.o.p.PluginsService] [opensearch-cluster-data-0] loaded plugin [opensearch-sql]
[2022-08-12T19:13:03,064][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2022-08-12T19:13:03,127][INFO][o.o.e.NodeEnvironment] [opensearch-cluster-data-0] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/sda1)]], net usable_space [22.9gb], net total_space [28.8gb], types [ext4]
[2022-08-12T19:13:03,127][INFO][o.o.e.NodeEnvironment] [opensearch-cluster-data-0] heap size [512mb], compressed ordinary object pointers [true]
[2022-08-12T19:13:03,160][INFO][o.o.n.Node] [opensearch-cluster-data-0] node name [opensearch-cluster-data-0], node ID [sMk_c3saSKqawVRxgELY2w], cluster name [opensearch-cluster], roles [map[master:false], map[ingest:true], map[data: true], map[remote_cluster_client:false]]
[2022-08-12T19:13:15,954][WARN][o.o.s.c.Salt] [opensearch-cluster-data-0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2022-08-12T19:13:16,218][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-cluster-data-0] Message routing enabled: true
[2022-08-12T19:13:16,315][INFO][o.o.s.f.SecurityFilter] [opensearch-cluster-data-0] <NONE> indices are made immutable.
[2022-08-12T19:13:17,515][INFO][o.o.a.b.ADCircuitBreakerService] [opensearch-cluster-data-0] Registered memory breaker.
[2022-08-12T19:13:18,833][INFO][o.o.m.c.b.MLCircuitBreakerService] [opensearch-cluster-data-0] Registered ML memory breaker.
[2022-08-12T19:13:20,051][INFO][o.o.t.NettyAllocator] [opensearch-cluster-data-0] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
[2022-08-12T19:13:20,358][INFO][o.o.d.DiscoveryModule] [opensearch-cluster-data-0] using discovery type [zen] and seed hosts providers [settings]
[2022-08-12T19:13:22,230][WARN][o.o.g.DanglingIndicesState] [opensearch-cluster-data-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-08-12T19:13:23,946][INFO][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-cluster-data-0] PerformanceAnalyzer Enabled: false
[2022-08-12T19:13:24,037][INFO][o.o.n.Node] [opensearch-cluster-data-0] initialized
[2022-08-12T19:13:24,038][INFO][o.o.n.Node] [opensearch-cluster-data-0] starting ...
[2022-08-12T19:13:24,333][INFO][o.o.t.TransportService] [opensearch-cluster-data-0] publish_address {172.17.0.4:9300}, bound_addresses {0.0.0.0:9300}
[2022-08-12T19:13:24,774][INFO][o.o.b.BootstrapChecks] [opensearch-cluster-data-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-08-12T19:13:54,829][WARN][o.o.n.Node] [opensearch-cluster-data-0] timed out while waiting for initial discovery state - timeout: 30s
[2022-08-12T19:13:54,845][INFO][o.o.h.AbstractHttpServerTransport] [opensearch-cluster-data-0] publish_address {172.17.0.4:9200}, bound_addresses {0.0.0.0:9200}
[2022-08-12T19:13:54,845][INFO][o.o.n.Node] [opensearch-cluster-data-0] started
[2022-08-12T19:13:54,845][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] Node started
[2022-08-12T19:13:54,846][INFO][o.o.s.c.ConfigurationRepository] [opensearch-cluster-data-0] Will attempt to create index .opendistro_security and default configs if they are absent
[2022-08-12T19:13:54,847][INFO][o.o.s.c.ConfigurationRepository] [opensearch-cluster-data-0] Background init thread started. Install default config?: true
[2022-08-12T19:13:54,848][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] 0 OpenSearch Security modules loaded so far: []
[2022-08-12T19:14:24,864][ERROR][o.o.s.c.ConfigurationRepository] [opensearch-cluster-data-0] Cannot apply default config (this is maybe not an error!)
org.opensearch.discovery.ClusterManagerNotDiscoveredException: null
	at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAction.java:305) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.2.0.jar:2.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T19:14:24,889][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-data-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T19:14:37,906][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-data-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T19:14:50,909][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-data-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]

```

---

<div class="post-metadata">

**Author:** ![ddodoo](https://avatars.discourse-cdn.com/v4/letter/d/bcef8e/32.png) [@ddodoo](https://forum.opensearch.org/u/ddodoo)\
**Post date:** [August 12, 2022, 10:44pm UTC](https://forum.opensearch.org/t/open-search-security-not-initialized/10537/20 "2022-08-12T22:44:00Z")

</div>

@pablo

**opensearch-cluster-client-0**

```auto
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
**************************************************************************
 **This tool will be deprecated in the next major release of OpenSearch**
 **https://github.com/opensearch-project/security/issues/1755**
 **************************************************************************
OpenSearch Security Demo Installer
 **Warning: Do not use on production or public reachable systems**
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-2.2.0
Detected OpenSearch Security Version: 2.2.0.0
tee: /usr/share/opensearch/config/opensearch.yml: Permission denied

Enabling OpenSearch Security Plugin
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.2.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.2.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2022-08-12T19:13:19,532][INFO][o.o.n.Node] [opensearch-cluster-client-0] version[2.2.0], pid[45], build[tar/b1017fa3b9a1c781d4f34ecee411e0cdf930a515/2022-08-09T02:27:25.256769336Z], OS[Linux/5.4.0-1089-azure/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.4/17.0.4+8]
[2022-08-12T19:13:19,538][INFO][o.o.n.Node] [opensearch-cluster-client-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[2022-08-12T19:13:19,538][INFO][o.o.n.Node] [opensearch-cluster-client-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-8961332606370530363, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx512M, -Xms512M, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2022-08-12T19:13:23,844][WARN][stderr] [opensearch-cluster-client-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2022-08-12T19:13:23,844][WARN][stderr] [opensearch-cluster-client-0] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2022-08-12T19:13:23,845][WARN][stderr] [opensearch-cluster-client-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2022-08-12T19:13:23,929][INFO][o.o.s.s.t.SSLConfig] [opensearch-cluster-client-0] SSL dual mode is disabled
[2022-08-12T19:13:23,930][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] OpenSearch Config path is /usr/share/opensearch/config
[2022-08-12T19:13:24,746][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-client-0] JVM supports TLSv1.3
[2022-08-12T19:13:24,748][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-client-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2022-08-12T19:13:26,031][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-client-0] TLS Transport Client Provider : JDK
[2022-08-12T19:13:26,031][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-client-0] TLS Transport Server Provider : JDK
[2022-08-12T19:13:26,031][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-client-0] TLS HTTP Provider : JDK
[2022-08-12T19:13:26,031][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-client-0] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2022-08-12T19:13:26,031][INFO][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-client-0] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[2022-08-12T19:13:26,756][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] Clustername: opensearch-cluster
[2022-08-12T19:13:26,763][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[2022-08-12T19:13:26,763][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[2022-08-12T19:13:26,763][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] File /usr/share/opensearch/config/kirk.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:26,763][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:26,763][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] File /usr/share/opensearch/config/esnode-key.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:26,764][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:26,764][WARN][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] File /usr/share/opensearch/config/kirk-key.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:28,017][INFO][o.o.p.c.PluginSettings] [opensearch-cluster-client-0] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2022-08-12T19:13:29,219][INFO][o.o.i.r.ReindexPlugin] [opensearch-cluster-client-0] ReindexPlugin reloadSPI called
[2022-08-12T19:13:29,226][INFO][o.o.i.r.ReindexPlugin] [opensearch-cluster-client-0] Unable to find any implementation for RemoteReindexExtension
[2022-08-12T19:13:29,336][INFO][o.o.j.JobSchedulerPlugin] [opensearch-cluster-client-0] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2022-08-12T19:13:29,431][INFO][o.o.j.JobSchedulerPlugin] [opensearch-cluster-client-0] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2022-08-12T19:13:29,433][INFO][o.o.j.JobSchedulerPlugin] [opensearch-cluster-client-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-08-12T19:13:29,440][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [aggs-matrix-stats]
[2022-08-12T19:13:29,440][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [analysis-common]
[2022-08-12T19:13:29,440][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [geo]
[2022-08-12T19:13:29,440][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [ingest-common]
[2022-08-12T19:13:29,441][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [ingest-geoip]
[2022-08-12T19:13:29,441][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [ingest-user-agent]
[2022-08-12T19:13:29,441][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [lang-expression]
[2022-08-12T19:13:29,441][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [lang-mustache]
[2022-08-12T19:13:29,441][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [lang-painless]
[2022-08-12T19:13:29,441][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [mapper-extras]
[2022-08-12T19:13:29,441][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [opensearch-dashboards]
[2022-08-12T19:13:29,441][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [parent-join]
[2022-08-12T19:13:29,441][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [percolator]
[2022-08-12T19:13:29,441][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [rank-eval]
[2022-08-12T19:13:29,441][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [reindex]
[2022-08-12T19:13:29,441][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [repository-url]
[2022-08-12T19:13:29,441][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [systemd]
[2022-08-12T19:13:29,441][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded module [transport-netty4]
[2022-08-12T19:13:29,442][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-alerting]
[2022-08-12T19:13:29,442][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-anomaly-detection]
[2022-08-12T19:13:29,442][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-asynchronous-search]
[2022-08-12T19:13:29,442][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-cross-cluster-replication]
[2022-08-12T19:13:29,442][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-geospatial]
[2022-08-12T19:13:29,442][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-index-management]
[2022-08-12T19:13:29,442][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-job-scheduler]
[2022-08-12T19:13:29,442][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-knn]
[2022-08-12T19:13:29,443][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-ml]
[2022-08-12T19:13:29,443][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-notifications]
[2022-08-12T19:13:29,443][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-notifications-core]
[2022-08-12T19:13:29,443][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-observability]
[2022-08-12T19:13:29,443][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-performance-analyzer]
[2022-08-12T19:13:29,443][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-reports-scheduler]
[2022-08-12T19:13:29,443][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-security]
[2022-08-12T19:13:29,443][INFO][o.o.p.PluginsService] [opensearch-cluster-client-0] loaded plugin [opensearch-sql]
[2022-08-12T19:13:29,527][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2022-08-12T19:13:29,558][INFO][o.o.e.NodeEnvironment] [opensearch-cluster-client-0] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/sda1)]], net usable_space [22.9gb], net total_space [28.8gb], types [ext4]
[2022-08-12T19:13:29,559][INFO][o.o.e.NodeEnvironment] [opensearch-cluster-client-0] heap size [512mb], compressed ordinary object pointers [true]
[2022-08-12T19:13:29,660][INFO][o.o.n.Node] [opensearch-cluster-client-0] node name [opensearch-cluster-client-0], node ID [aQ165eaKQ4iFBoQS7gwvAQ], cluster name [opensearch-cluster], roles [map[master:false], map[data:false], map[ingest:false], map[remote_cluster_client:false]]
[2022-08-12T19:13:36,229][WARN][o.o.s.c.Salt] [opensearch-cluster-client-0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2022-08-12T19:13:36,325][INFO][o.o.s.a.i.AuditLogImpl] [opensearch-cluster-client-0] Message routing enabled: true
[2022-08-12T19:13:36,367][INFO][o.o.s.f.SecurityFilter] [opensearch-cluster-client-0] <NONE> indices are made immutable.
[2022-08-12T19:13:37,089][INFO][o.o.a.b.ADCircuitBreakerService] [opensearch-cluster-client-0] Registered memory breaker.
[2022-08-12T19:13:37,754][INFO][o.o.m.c.b.MLCircuitBreakerService] [opensearch-cluster-client-0] Registered ML memory breaker.
[2022-08-12T19:13:38,299][INFO][o.o.t.NettyAllocator] [opensearch-cluster-client-0] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
[2022-08-12T19:13:38,420][INFO][o.o.d.DiscoveryModule] [opensearch-cluster-client-0] using discovery type [zen] and seed hosts providers [settings]
[2022-08-12T19:13:39,033][WARN][o.o.g.DanglingIndicesState] [opensearch-cluster-client-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-08-12T19:13:39,607][INFO][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-cluster-client-0] PerformanceAnalyzer Enabled: false
[2022-08-12T19:13:39,648][INFO][o.o.n.Node] [opensearch-cluster-client-0] initialized
[2022-08-12T19:13:39,648][INFO][o.o.n.Node] [opensearch-cluster-client-0] starting ...
[2022-08-12T19:13:39,800][INFO][o.o.t.TransportService] [opensearch-cluster-client-0] publish_address {172.17.0.5:9300}, bound_addresses {0.0.0.0:9300}
[2022-08-12T19:13:40,052][INFO][o.o.b.BootstrapChecks] [opensearch-cluster-client-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-08-12T19:14:10,089][WARN][o.o.n.Node] [opensearch-cluster-client-0] timed out while waiting for initial discovery state - timeout: 30s
[2022-08-12T19:14:10,122][INFO][o.o.h.AbstractHttpServerTransport] [opensearch-cluster-client-0] publish_address {172.17.0.5:9200}, bound_addresses {0.0.0.0:9200}
[2022-08-12T19:14:10,123][INFO][o.o.n.Node] [opensearch-cluster-client-0] started
[2022-08-12T19:14:10,124][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] Node started
[2022-08-12T19:14:10,124][INFO][o.o.s.c.ConfigurationRepository] [opensearch-cluster-client-0] Will attempt to create index .opendistro_security and default configs if they are absent
[2022-08-12T19:14:10,128][INFO][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] 0 OpenSearch Security modules loaded so far: []
[2022-08-12T19:14:10,130][INFO][o.o.s.c.ConfigurationRepository] [opensearch-cluster-client-0] Background init thread started. Install default config?: true
[2022-08-12T19:14:40,143][ERROR][o.o.s.c.ConfigurationRepository] [opensearch-cluster-client-0] Cannot apply default config (this is maybe not an error!)
org.opensearch.discovery.ClusterManagerNotDiscoveredException: null
	at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAction.java:305) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.2.0.jar:2.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T19:14:40,160][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-client-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T19:14:53,179][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-client-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]

```

[Next page](https://forum.opensearch.org/t/open-search-security-not-initialized/10537.md?page=2)