Open Search Security Not Initialized

ddodoo · August 8, 2022, 4:31pm

I run into an error when excecuting the securityadmin.sh script from the Documentation

./securityadmin.sh -cd …/…/…/config/opensearch-security/ -icl -nhnv
-cacert …/…/…/config/root-ca.pem
-cert …/…/…/config/kirk.pem
-key …/…/…/config/kirk-key.pem

I receive the following error response :

Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates) 
Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml 
If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

Can you help me troubleshoot the possible cause?

Kindly find my log file attached:
securityadmin_diag_trace

nateynate · August 8, 2022, 5:26pm

Hi @ddodoo -

I’m noticing two things here that I’d like to verify before I jump to conclusions. First, it looks like you’re invoking securityadmin.sh but passing references to files using a series of three dots instead of two. Is this intentional, or just a result of pasting the output into the forum here?

Secondly, I’m seeing this in the trace:

OpenSearchStatusException[OpenSearch exception [type=master_not_discovered_exception, reason=null]]
	at org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:207)
	at org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2081)
	at org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2058)
	at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1777)
	at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1730)
	at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1698)
	at org.opensearch.client.ClusterClient.health(ClusterClient.java:162)
	at org.opensearch.security.tools.SecurityAdmin.generateDiagnoseTrace(SecurityAdmin.java:999)
	at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:595)
	at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:159)
	Suppressed: org.opensearch.client.ResponseException: method [GET], host [https://localhost:9200], URI [/_cluster/health?master_timeout=30s&level=cluster&timeout=30s], status line [HTTP/1.1 503 Service Unavailable]
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
		at org.opensearch.client.RestClient.convertResponse(RestClient.java:350)
		at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
		at org.opensearch.client.RestClient.performRequest(RestClient.java:295)
		at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764)
		... 6 more

I’m a bit out of my element here, but let’s try without the triple dots.

Nate

ddodoo · August 8, 2022, 5:43pm

Hello @nateynate , I examined both the original documentation and the script I executed in the terminal

It has 2 dots and not 3

I guess the 3 dots is from the forum page formatting

pablo · August 8, 2022, 6:08pm

@ddodoo According to the logs, service at port 9200 is not up.
Do you see any other errors? Did your nodes initialize?

Please share your config.yml and opensearch.yml files.
How many nodes do you have in the cluster?

Also, please share the output of the below command.

curl --insecure -u admin:admin -XGET https://localhost:9200/

ddodoo · August 9, 2022, 7:12am

I followed the port-forward command from the OpenSearch Multi-Node Cluster Kubernetes documentation @pablo

Setup OpenSearch multi-node cluster on Kubernetes using Helm Charts · OpenSearch

 kubectl port-forward opensearch-cluster-master-0 9200

I have a single node

The output of the curl --insecure -u admin:admin -XGET https://localhost:9200/
is “OpenSearch Security not initialized”

Kindly find the image attached :

Kindly find the config.yml and opensearch.yml files below:

config.yml

---

# This is the main OpenSearch Security configuration file where authentication
# and authorization is defined.
#
# You need to configure at least one authentication domain in the authc of this file.
# An authentication domain is responsible for extracting the user credentials from
# the request and for validating them against an authentication backend like Active Directory for example.
#
# If more than one authentication domain is configured the first one which succeeds wins.
# If all authentication domains fail then the request is unauthenticated.
# In this case an exception is thrown and/or the HTTP status is set to 401.
#
# After authentication authorization (authz) will be applied. There can be zero or more authorizers which collect
# the roles from a given backend for the authenticated user.
#
# Both, authc and auth can be enabled/disabled separately for REST and TRANSPORT layer. Default is true for both.
#        http_enabled: true
#        transport_enabled: true
#
# For HTTP it is possible to allow anonymous authentication. If that is the case then the HTTP authenticators try to
# find user credentials in the HTTP request. If credentials are found then the user gets regularly authenticated.
# If none can be found the user will be authenticated as an "anonymous" user. This user has always the username "anonymous"
# and one role named "anonymous_backendrole".
# If you enable anonymous authentication all HTTP authenticators will not challenge.
#
#
# Note: If you define more than one HTTP authenticators make sure to put non-challenging authenticators like "proxy" or "clientcert"
# first and the challenging one last.
# Because it's not possible to challenge a client with two different authentication methods (for example
# Kerberos and Basic) only one can have the challenge flag set to true. You can cope with this situation
# by using pre-authentication, e.g. sending a HTTP Basic authentication header in the request.
#
# Default value of the challenge flag is true.
#
#
# HTTP
#   basic (challenging)
#   proxy (not challenging, needs xff)
#   kerberos (challenging)
#   clientcert (not challenging, needs https)
#   jwt (not challenging)
#   host (not challenging) #DEPRECATED, will be removed in a future version.
#                          host based authentication is configurable in roles_mapping

# Authc
#   internal
#   noop
#   ldap

# Authz
#   ldap
#   noop



_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
    # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
    # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
    #filtered_alias_mode: warn
    #do_not_fail_on_forbidden: false
    #kibana:
    # Kibana multitenancy
    #multitenancy_enabled: true
    #server_username: kibanaserver
    #index: '.kibana'
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
        #internalProxies: '.*' # trust all internal proxies, regex pattern
        #remoteIpHeader:  'x-forwarded-for'
        ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
        ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
        ###### and here https://tools.ietf.org/html/rfc7239
        ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
    authc:
      kerberos_auth_domain:
        http_enabled: false
        transport_enabled: false
        order: 6
        http_authenticator:
          type: kerberos
          challenge: true
          config:
            # If true a lot of kerberos/security related debugging output will be logged to standard out
            krb_debug: false
            # If true then the realm will be stripped from the user name
            strip_realm_from_principal: true
        authentication_backend:
          type: noop
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 4
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
      proxy_auth_domain:
        description: "Authenticate via proxy"
        http_enabled: false
        transport_enabled: false
        order: 3
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: "x-proxy-user"
            roles_header: "x-proxy-roles"
        authentication_backend:
          type: noop
      jwt_auth_domain:
        description: "Authenticate via Json Web Token"
        http_enabled: false
        transport_enabled: false
        order: 0
        http_authenticator:
          type: jwt
          challenge: false
          config:
            signing_key: "base64 encoded HMAC key or public RSA/ECDSA pem key"
            jwt_header: "Authorization"
            jwt_url_parameter: null
            roles_key: null
            subject_key: null
        authentication_backend:
          type: noop
      clientcert_auth_domain:
        description: "Authenticate via SSL client certificates"
        http_enabled: false
        transport_enabled: false
        order: 2
        http_authenticator:
          type: clientcert
          config:
            username_attribute: cn #optional, if omitted DN becomes username
          challenge: false
        authentication_backend:
          type: noop
      ldap:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        order: 5
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          # LDAP authentication backend (authenticate users against a LDAP or Active Directory)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - localhost:8389
            bind_dn: null
            password: null
            userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(sAMAccountName={0})'
            # Use this attribute from the user as username (if not set then DN is used)
            username_attribute: null
    authz:
      roles_from_myldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: false
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            hosts:
            - localhost:8389
            bind_dn: null
            password: null
            rolebase: 'ou=groups,dc=example,dc=com'
            # Filter to search for roles (currently in the whole subtree beneath rolebase)
            # {0} is substituted with the DN of the user
            # {1} is substituted with the username
            # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
            rolesearch: '(member={0})'
            # Specify the name of the attribute which value should be substituted with {2} above
            userroleattribute: null
            # Roles as an attribute of the user entry
            userrolename: disabled
            #userrolename: memberOf
            # The attribute in a role entry containing the name of that role, Default is "name".
            # Can also be "dn" to use the full DN as rolename.
            rolename: cn
            # Resolve nested roles transitive (roles which are members of other roles and so on ...)
            resolve_nested_roles: true
            userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(uid={0})'
            # Skip users matching a user name, a wildcard or a regex pattern
            #skip_users:
            #  - 'cn=Michael Jackson,ou*people,o=TEST'
            #  - '/\S*/'
      roles_from_another_ldap:
        description: "Authorize via another Active Directory"
        http_enabled: false
        transport_enabled: false
        authorization_backend:
          type: ldap
          #config goes here ...
  #    auth_failure_listeners:
  #      ip_rate_limiting:
  #        type: ip
  #        allowed_tries: 10
  #        time_window_seconds: 3600
  #        block_expiry_seconds: 600
  #        max_blocked_clients: 100000
  #        max_tracked_clients: 100000
  #      internal_authentication_backend_limiting:
  #        type: username
  #        authentication_backend: intern
  #        allowed_tries: 10
  #        time_window_seconds: 3600
  #        block_expiry_seconds: 600
  #        max_blocked_clients: 100000
  #        max_tracked_clients: 100000

opensearch.yml

cluster.name: opensearch-cluster

# Bind to all interfaces because we don't know what IP address Docker will assign to us.
network.host: 0.0.0.0

# Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
# discovery.type: single-node

# Start OpenSearch Security Demo Configuration
# WARNING: revise all the lines below before you go into production
plugins:
  security:
    ssl:
      transport:
        pemcert_filepath: esnode.pem
        pemkey_filepath: esnode-key.pem
        pemtrustedcas_filepath: root-ca.pem
        enforce_hostname_verification: false
      http:
        enabled: true
        pemcert_filepath: esnode.pem
        pemkey_filepath: esnode-key.pem
        pemtrustedcas_filepath: root-ca.pem
    allow_unsafe_democertificates: true
    allow_default_init_securityindex: true
    authcz:
      admin_dn:
        - CN=kirk,OU=client,O=client,L=test,C=de
    audit.type: internal_opensearch
    enable_snapshot_restore_privilege: true
    check_snapshot_restore_write_privileges: true
    restapi:
      roles_enabled: ["all_access", "security_rest_api_access"]
    system_indices:
      enabled: true
      indices:
        [
          ".opendistro-alerting-config",
          ".opendistro-alerting-alert*",
          ".opendistro-anomaly-results*",
          ".opendistro-anomaly-detector*",
          ".opendistro-anomaly-checkpoints",
          ".opendistro-anomaly-detection-state",
          ".opendistro-reports-*",
          ".opendistro-notifications-*",
          ".opendistro-notebooks",
          ".opendistro-asynchronous-search-response*",
        ]
######## End OpenSearch Security Demo Configuration ########

ddodoo · August 9, 2022, 7:14am

Port Forwarding

@pablo

ddodoo · August 9, 2022, 7:15am

Executing securityadmin.sh Script

@pablo

pablo · August 9, 2022, 1:03pm

@ddodoo @ddodoo I understand that you’ve followed Setup OpenSearch multi-node cluster on Kubernetes using Helm Charts and you have created a single-node cluster and not multi-node as described in the setup.

According to your screenshots, you run opensearch-cluster-master-0 pod. Did you follow master.yaml config and set the below values?

If so, then your cluster doesn’t have any node with data role. Could you confirm that?
Also, did you set discovery.type: to single-node

ddodoo · August 10, 2022, 2:02pm

Yes I have the same configuration to my master.yaml file @pablo

How do I set discovery.type: to single-node?

# These will be set as environment variable "node. Roles". E.g. node.roles=master,ingest,data,remote_cluster_client
roles:
  - master: "true"
  - ingest: "false"
  - data: "false"
  - remote_cluster_client: "false"

replicas: 1

pablo · August 10, 2022, 2:28pm

@ddodoo The main problem is in the roles. If you’ve just used master.yaml then that master node has no data role. Without a data role, no indices, including .opendistro_security, will be created.
You should be already seeing errors in the OpenSearch pod’s logs regarding the security configuration.

The example described in the manual creates 3 nodes cluster.

The quick fix is to set all the flags to true on your single master node.

This should work without setting single-node. If not, try setting it in the opensearch.yml section in master.yaml file.

ddodoo · August 10, 2022, 4:32pm

I used helm to install all 3 yaml files (master.yaml, client.yaml and data.yaml) as suggested in the documentation and not just the master.yaml file

@pablo

pauchamb · August 10, 2022, 4:40pm

Hey @ddodoo

I ran into a similar issue like this. The difference between our environments is that I installed OpenSearch via an RPM but the two settings I needed when first starting up my cluster were:

plugins.security.allow_default_init_securityindex: true
cluster.initial_cluster_manager_nodes: [ list, of, nodes ]

Once I had those in opensearch.yml and restarted, the cluster initialized just fine with any number of nodes.

ddodoo · August 10, 2022, 6:29pm

Is this already in the opensearch.yml file or I have to add it? @pauchamb

plugins.security.allow_default_init_securityindex: true
cluster.initial_cluster_manager_nodes: [ list, of, nodes ]

pablo · August 11, 2022, 10:38am

@ddodoo Could you share the output of the below command?

kubectl get all -A

ddodoo · August 11, 2022, 9:59pm

Sure sure

Kindly view the response @pablo

root@ubuntu-test-vm:/home/azureuser# kubectl get all -A
NAMESPACE     NAME                                                   READY   STATUS    RESTARTS      AGE
default       pod/dashboards-opensearch-dashboards-584c75fcb-zfqld   1/1     Running   0             21m
default       pod/opensearch-cluster-client-0                        1/1     Running   0             31m
default       pod/opensearch-cluster-data-0                          1/1     Running   0             32m
default       pod/opensearch-cluster-master-0                        1/1     Running   0             32m
kube-system   pod/coredns-6d4b75cb6d-p9zwq                           1/1     Running   0             37m
kube-system   pod/etcd-minikube                                      1/1     Running   0             37m
kube-system   pod/kube-apiserver-minikube                            1/1     Running   0             37m
kube-system   pod/kube-controller-manager-minikube                   1/1     Running   0             37m
kube-system   pod/kube-proxy-vjstv                                   1/1     Running   0             37m
kube-system   pod/kube-scheduler-minikube                            1/1     Running   0             37m
kube-system   pod/storage-provisioner                                1/1     Running   1 (36m ago)   37m

NAMESPACE     NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
default       service/dashboards-opensearch-dashboards     ClusterIP   10.96.13.93      <none>        5601/TCP                 21m
default       service/kubernetes                           ClusterIP   10.96.0.1        <none>        443/TCP                  37m
default       service/opensearch-cluster-client            ClusterIP   10.106.244.135   <none>        9200/TCP,9300/TCP        31m
default       service/opensearch-cluster-client-headless   ClusterIP   None             <none>        9200/TCP,9300/TCP        31m
default       service/opensearch-cluster-data              ClusterIP   10.109.206.142   <none>        9200/TCP,9300/TCP        32m
default       service/opensearch-cluster-data-headless     ClusterIP   None             <none>        9200/TCP,9300/TCP        32m
default       service/opensearch-cluster-master            ClusterIP   10.99.170.57     <none>        9200/TCP,9300/TCP        32m
default       service/opensearch-cluster-master-headless   ClusterIP   None             <none>        9200/TCP,9300/TCP        32m
kube-system   service/kube-dns                             ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP   37m

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   37m

NAMESPACE     NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
default       deployment.apps/dashboards-opensearch-dashboards   1/1     1            1           21m
kube-system   deployment.apps/coredns                            1/1     1            1           37m

NAMESPACE     NAME                                                         DESIRED   CURRENT   READY   AGE
default       replicaset.apps/dashboards-opensearch-dashboards-584c75fcb   1         1         1       21m
kube-system   replicaset.apps/coredns-6d4b75cb6d                           1         1         1       37m

NAMESPACE   NAME                                         READY   AGE
default     statefulset.apps/opensearch-cluster-client   1/1     31m
default     statefulset.apps/opensearch-cluster-data     1/1     32m
default     statefulset.apps/opensearch-cluster-master   1/1     32m

ddodoo · August 11, 2022, 10:01pm

I cannot make changes to the pod container because the opensearch.yml file is a read-only file?

How do I resolve this?

@pauchamb

pablo · August 12, 2022, 10:27am

@ddodoo Could you run the following commands and share the files and outputs?

Logs from each OpenSearch container

kubectl logs opensearch-cluster-client-0 > opensearch-cluster-client-0.log
kubectl logs opensearch-cluster-master-0 > opensearch-cluster-master-0.log
kubectl logs opensearch-cluster-data-0 > opensearch-cluster-data-0.log

Run port-forwarding for data node and run curl against that.

kubectl port-forward opensearch-cluster-data-0 9200

curl --insecure -u admin:admin -XGET https://localhost:9200/

ddodoo · August 12, 2022, 10:40pm

@pablo

opensearch-cluster-master-0

Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-2.2.0
Detected OpenSearch Security Version: 2.2.0.0

tee: /usr/share/opensearch/config/opensearch.yml: Permission denied
Enabling OpenSearch Security Plugin
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.2.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.2.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2022-08-12T17:18:38,737][INFO ][o.o.n.Node               ] [opensearch-cluster-master-0] version[2.2.0], pid[45], build[tar/b1017fa3b9a1c781d4f34ecee411e0cdf930a515/2022-08-09T02:27:25.256769336Z], OS[Linux/5.4.0-1089-azure/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.4/17.0.4+8]
[2022-08-12T17:18:38,739][INFO ][o.o.n.Node               ] [opensearch-cluster-master-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[2022-08-12T17:18:38,739][INFO ][o.o.n.Node               ] [opensearch-cluster-master-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-5998433139062988299, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx512M, -Xms512M, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2022-08-12T17:18:40,058][WARN ][stderr                   ] [opensearch-cluster-master-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2022-08-12T17:18:40,059][WARN ][stderr                   ] [opensearch-cluster-master-0] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2022-08-12T17:18:40,059][WARN ][stderr                   ] [opensearch-cluster-master-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2022-08-12T17:18:40,071][INFO ][o.o.s.s.t.SSLConfig      ] [opensearch-cluster-master-0] SSL dual mode is disabled
[2022-08-12T17:18:40,071][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] OpenSearch Config path is /usr/share/opensearch/config
[2022-08-12T17:18:40,470][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] JVM supports TLSv1.3
[2022-08-12T17:18:40,472][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2022-08-12T17:18:40,745][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] TLS Transport Client Provider : JDK
[2022-08-12T17:18:40,746][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] TLS Transport Server Provider : JDK
[2022-08-12T17:18:40,746][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] TLS HTTP Provider             : JDK
[2022-08-12T17:18:40,746][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2022-08-12T17:18:40,746][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2]
[2022-08-12T17:18:41,113][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] Clustername: opensearch-cluster
[2022-08-12T17:18:41,118][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[2022-08-12T17:18:41,118][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[2022-08-12T17:18:41,119][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/kirk.pem has insecure file permissions (should be 0600)
[2022-08-12T17:18:41,119][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
[2022-08-12T17:18:41,119][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/esnode-key.pem has insecure file permissions (should be 0600)
[2022-08-12T17:18:41,119][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600)
[2022-08-12T17:18:41,119][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/kirk-key.pem has insecure file permissions (should be 0600)
[2022-08-12T17:18:41,938][INFO ][o.o.p.c.PluginSettings   ] [opensearch-cluster-master-0] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2022-08-12T17:18:42,639][INFO ][o.o.i.r.ReindexPlugin    ] [opensearch-cluster-master-0] ReindexPlugin reloadSPI called
[2022-08-12T17:18:42,640][INFO ][o.o.i.r.ReindexPlugin    ] [opensearch-cluster-master-0] Unable to find any implementation for RemoteReindexExtension
[2022-08-12T17:18:42,714][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2022-08-12T17:18:42,741][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2022-08-12T17:18:42,742][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-08-12T17:18:42,748][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [aggs-matrix-stats]
[2022-08-12T17:18:42,748][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [analysis-common]
[2022-08-12T17:18:42,748][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [geo]
[2022-08-12T17:18:42,748][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [ingest-common]
[2022-08-12T17:18:42,748][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [ingest-geoip]
[2022-08-12T17:18:42,748][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [ingest-user-agent]
[2022-08-12T17:18:42,748][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [lang-expression]
[2022-08-12T17:18:42,748][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [lang-mustache]
[2022-08-12T17:18:42,748][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [lang-painless]
[2022-08-12T17:18:42,748][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [mapper-extras]
[2022-08-12T17:18:42,748][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [opensearch-dashboards]
[2022-08-12T17:18:42,748][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [parent-join]
[2022-08-12T17:18:42,748][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [percolator]
[2022-08-12T17:18:42,748][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [rank-eval]
[2022-08-12T17:18:42,749][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [reindex]
[2022-08-12T17:18:42,749][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [repository-url]
[2022-08-12T17:18:42,749][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [systemd]
[2022-08-12T17:18:42,749][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded module [transport-netty4]
[2022-08-12T17:18:42,749][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-alerting]
[2022-08-12T17:18:42,749][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-anomaly-detection]
[2022-08-12T17:18:42,749][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-asynchronous-search]
[2022-08-12T17:18:42,750][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-cross-cluster-replication]
[2022-08-12T17:18:42,750][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-geospatial]
[2022-08-12T17:18:42,750][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-index-management]
[2022-08-12T17:18:42,750][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-job-scheduler]
[2022-08-12T17:18:42,750][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-knn]
[2022-08-12T17:18:42,750][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-ml]
[2022-08-12T17:18:42,750][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-notifications]
[2022-08-12T17:18:42,750][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-notifications-core]
[2022-08-12T17:18:42,750][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-observability]
[2022-08-12T17:18:42,750][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-performance-analyzer]
[2022-08-12T17:18:42,750][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-reports-scheduler]
[2022-08-12T17:18:42,750][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-security]
[2022-08-12T17:18:42,750][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-master-0] loaded plugin [opensearch-sql]
[2022-08-12T17:18:42,768][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2022-08-12T17:18:42,827][INFO ][o.o.e.NodeEnvironment    ] [opensearch-cluster-master-0] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/sda1)]], net usable_space [22.9gb], net total_space [28.8gb], types [ext4]
[2022-08-12T17:18:42,827][INFO ][o.o.e.NodeEnvironment    ] [opensearch-cluster-master-0] heap size [512mb], compressed ordinary object pointers [true]
[2022-08-12T17:18:42,863][INFO ][o.o.n.Node               ] [opensearch-cluster-master-0] node name [opensearch-cluster-master-0], node ID [R2a0BBXMRQ6fpE7RO050dw], cluster name [opensearch-cluster], roles [map[data:false], map[ingest:false], map[master:true], map[remote_cluster_client:false]]
[2022-08-12T17:18:48,253][WARN ][o.o.s.c.Salt             ] [opensearch-cluster-master-0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2022-08-12T17:18:48,326][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-cluster-master-0] Message routing enabled: true
[2022-08-12T17:18:48,364][INFO ][o.o.s.f.SecurityFilter   ] [opensearch-cluster-master-0] <NONE> indices are made immutable.
[2022-08-12T17:18:48,863][INFO ][o.o.a.b.ADCircuitBreakerService] [opensearch-cluster-master-0] Registered memory breaker.
[2022-08-12T17:18:49,324][INFO ][o.o.m.c.b.MLCircuitBreakerService] [opensearch-cluster-master-0] Registered ML memory breaker.
[2022-08-12T17:18:49,915][INFO ][o.o.t.NettyAllocator     ] [opensearch-cluster-master-0] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
[2022-08-12T17:18:50,012][INFO ][o.o.d.DiscoveryModule    ] [opensearch-cluster-master-0] using discovery type [zen] and seed hosts providers [settings]
[2022-08-12T17:18:50,622][WARN ][o.o.g.DanglingIndicesState] [opensearch-cluster-master-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-08-12T17:18:51,313][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-cluster-master-0] PerformanceAnalyzer Enabled: false
[2022-08-12T17:18:51,342][INFO ][o.o.n.Node               ] [opensearch-cluster-master-0] initialized
[2022-08-12T17:18:51,342][INFO ][o.o.n.Node               ] [opensearch-cluster-master-0] starting ...
[2022-08-12T17:18:51,428][INFO ][o.o.t.TransportService   ] [opensearch-cluster-master-0] publish_address {172.17.0.3:9300}, bound_addresses {0.0.0.0:9300}
[2022-08-12T17:18:51,596][INFO ][o.o.b.BootstrapChecks    ] [opensearch-cluster-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-08-12T17:19:21,612][WARN ][o.o.n.Node               ] [opensearch-cluster-master-0] timed out while waiting for initial discovery state - timeout: 30s
[2022-08-12T17:19:21,619][INFO ][o.o.h.AbstractHttpServerTransport] [opensearch-cluster-master-0] publish_address {172.17.0.3:9200}, bound_addresses {0.0.0.0:9200}
[2022-08-12T17:19:21,619][INFO ][o.o.n.Node               ] [opensearch-cluster-master-0] started
[2022-08-12T17:19:21,620][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] Node started
[2022-08-12T17:19:21,620][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-master-0] Will attempt to create index .opendistro_security and default configs if they are absent
[2022-08-12T17:19:21,621][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-master-0] Background init thread started. Install default config?: true
[2022-08-12T17:19:21,621][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] 0 OpenSearch Security modules loaded so far: []
[2022-08-12T17:19:51,628][ERROR][o.o.s.c.ConfigurationRepository] [opensearch-cluster-master-0] Cannot apply default config (this is maybe not an error!)
org.opensearch.discovery.ClusterManagerNotDiscoveredException: null
	at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAction.java:305) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.2.0.jar:2.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T17:19:51,634][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-master-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T17:20:04,638][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-master-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T17:20:17,640][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-master-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]

ddodoo · August 12, 2022, 10:41pm

@pablo

opensearch-cluster-data-0

Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-2.2.0
Detected OpenSearch Security Version: 2.2.0.0
tee: /usr/share/opensearch/config/opensearch.yml: Permission denied

Enabling OpenSearch Security Plugin
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.2.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.2.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2022-08-12T19:12:58,240][INFO ][o.o.n.Node               ] [opensearch-cluster-data-0] version[2.2.0], pid[44], build[tar/b1017fa3b9a1c781d4f34ecee411e0cdf930a515/2022-08-09T02:27:25.256769336Z], OS[Linux/5.4.0-1089-azure/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.4/17.0.4+8]
[2022-08-12T19:12:58,243][INFO ][o.o.n.Node               ] [opensearch-cluster-data-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[2022-08-12T19:12:58,243][INFO ][o.o.n.Node               ] [opensearch-cluster-data-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-3276608203265200185, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx512M, -Xms512M, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2022-08-12T19:12:59,716][WARN ][stderr                   ] [opensearch-cluster-data-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2022-08-12T19:12:59,716][WARN ][stderr                   ] [opensearch-cluster-data-0] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2022-08-12T19:12:59,717][WARN ][stderr                   ] [opensearch-cluster-data-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2022-08-12T19:12:59,732][INFO ][o.o.s.s.t.SSLConfig      ] [opensearch-cluster-data-0] SSL dual mode is disabled
[2022-08-12T19:12:59,732][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] OpenSearch Config path is /usr/share/opensearch/config
[2022-08-12T19:13:00,257][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-data-0] JVM supports TLSv1.3
[2022-08-12T19:13:00,260][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-data-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2022-08-12T19:13:00,944][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-data-0] TLS Transport Client Provider : JDK
[2022-08-12T19:13:00,944][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-data-0] TLS Transport Server Provider : JDK
[2022-08-12T19:13:00,944][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-data-0] TLS HTTP Provider             : JDK
[2022-08-12T19:13:00,944][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-data-0] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2022-08-12T19:13:00,944][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-data-0] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2]
[2022-08-12T19:13:01,367][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] Clustername: opensearch-cluster
[2022-08-12T19:13:01,417][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[2022-08-12T19:13:01,418][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[2022-08-12T19:13:01,418][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] File /usr/share/opensearch/config/kirk.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:01,418][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:01,418][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] File /usr/share/opensearch/config/esnode-key.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:01,418][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:01,418][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] File /usr/share/opensearch/config/kirk-key.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:02,134][INFO ][o.o.p.c.PluginSettings   ] [opensearch-cluster-data-0] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2022-08-12T19:13:02,929][INFO ][o.o.i.r.ReindexPlugin    ] [opensearch-cluster-data-0] ReindexPlugin reloadSPI called
[2022-08-12T19:13:02,930][INFO ][o.o.i.r.ReindexPlugin    ] [opensearch-cluster-data-0] Unable to find any implementation for RemoteReindexExtension
[2022-08-12T19:13:03,013][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-data-0] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2022-08-12T19:13:03,037][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-data-0] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2022-08-12T19:13:03,038][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-data-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-08-12T19:13:03,044][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [aggs-matrix-stats]
[2022-08-12T19:13:03,044][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [analysis-common]
[2022-08-12T19:13:03,044][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [geo]
[2022-08-12T19:13:03,044][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [ingest-common]
[2022-08-12T19:13:03,044][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [ingest-geoip]
[2022-08-12T19:13:03,044][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [ingest-user-agent]
[2022-08-12T19:13:03,044][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [lang-expression]
[2022-08-12T19:13:03,044][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [lang-mustache]
[2022-08-12T19:13:03,044][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [lang-painless]
[2022-08-12T19:13:03,044][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [mapper-extras]
[2022-08-12T19:13:03,044][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [opensearch-dashboards]
[2022-08-12T19:13:03,045][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [parent-join]
[2022-08-12T19:13:03,045][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [percolator]
[2022-08-12T19:13:03,045][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [rank-eval]
[2022-08-12T19:13:03,045][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [reindex]
[2022-08-12T19:13:03,045][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [repository-url]
[2022-08-12T19:13:03,045][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [systemd]
[2022-08-12T19:13:03,045][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded module [transport-netty4]
[2022-08-12T19:13:03,045][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-alerting]
[2022-08-12T19:13:03,045][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-anomaly-detection]
[2022-08-12T19:13:03,046][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-asynchronous-search]
[2022-08-12T19:13:03,046][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-cross-cluster-replication]
[2022-08-12T19:13:03,046][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-geospatial]
[2022-08-12T19:13:03,046][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-index-management]
[2022-08-12T19:13:03,046][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-job-scheduler]
[2022-08-12T19:13:03,046][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-knn]
[2022-08-12T19:13:03,046][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-ml]
[2022-08-12T19:13:03,046][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-notifications]
[2022-08-12T19:13:03,046][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-notifications-core]
[2022-08-12T19:13:03,046][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-observability]
[2022-08-12T19:13:03,046][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-performance-analyzer]
[2022-08-12T19:13:03,046][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-reports-scheduler]
[2022-08-12T19:13:03,046][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-security]
[2022-08-12T19:13:03,046][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-data-0] loaded plugin [opensearch-sql]
[2022-08-12T19:13:03,064][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2022-08-12T19:13:03,127][INFO ][o.o.e.NodeEnvironment    ] [opensearch-cluster-data-0] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/sda1)]], net usable_space [22.9gb], net total_space [28.8gb], types [ext4]
[2022-08-12T19:13:03,127][INFO ][o.o.e.NodeEnvironment    ] [opensearch-cluster-data-0] heap size [512mb], compressed ordinary object pointers [true]
[2022-08-12T19:13:03,160][INFO ][o.o.n.Node               ] [opensearch-cluster-data-0] node name [opensearch-cluster-data-0], node ID [sMk_c3saSKqawVRxgELY2w], cluster name [opensearch-cluster], roles [map[master:false], map[ingest:true], map[data: true], map[remote_cluster_client:false]]
[2022-08-12T19:13:15,954][WARN ][o.o.s.c.Salt             ] [opensearch-cluster-data-0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2022-08-12T19:13:16,218][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-cluster-data-0] Message routing enabled: true
[2022-08-12T19:13:16,315][INFO ][o.o.s.f.SecurityFilter   ] [opensearch-cluster-data-0] <NONE> indices are made immutable.
[2022-08-12T19:13:17,515][INFO ][o.o.a.b.ADCircuitBreakerService] [opensearch-cluster-data-0] Registered memory breaker.
[2022-08-12T19:13:18,833][INFO ][o.o.m.c.b.MLCircuitBreakerService] [opensearch-cluster-data-0] Registered ML memory breaker.
[2022-08-12T19:13:20,051][INFO ][o.o.t.NettyAllocator     ] [opensearch-cluster-data-0] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
[2022-08-12T19:13:20,358][INFO ][o.o.d.DiscoveryModule    ] [opensearch-cluster-data-0] using discovery type [zen] and seed hosts providers [settings]
[2022-08-12T19:13:22,230][WARN ][o.o.g.DanglingIndicesState] [opensearch-cluster-data-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-08-12T19:13:23,946][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-cluster-data-0] PerformanceAnalyzer Enabled: false
[2022-08-12T19:13:24,037][INFO ][o.o.n.Node               ] [opensearch-cluster-data-0] initialized
[2022-08-12T19:13:24,038][INFO ][o.o.n.Node               ] [opensearch-cluster-data-0] starting ...
[2022-08-12T19:13:24,333][INFO ][o.o.t.TransportService   ] [opensearch-cluster-data-0] publish_address {172.17.0.4:9300}, bound_addresses {0.0.0.0:9300}
[2022-08-12T19:13:24,774][INFO ][o.o.b.BootstrapChecks    ] [opensearch-cluster-data-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-08-12T19:13:54,829][WARN ][o.o.n.Node               ] [opensearch-cluster-data-0] timed out while waiting for initial discovery state - timeout: 30s
[2022-08-12T19:13:54,845][INFO ][o.o.h.AbstractHttpServerTransport] [opensearch-cluster-data-0] publish_address {172.17.0.4:9200}, bound_addresses {0.0.0.0:9200}
[2022-08-12T19:13:54,845][INFO ][o.o.n.Node               ] [opensearch-cluster-data-0] started
[2022-08-12T19:13:54,845][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] Node started
[2022-08-12T19:13:54,846][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-data-0] Will attempt to create index .opendistro_security and default configs if they are absent
[2022-08-12T19:13:54,847][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-data-0] Background init thread started. Install default config?: true
[2022-08-12T19:13:54,848][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-data-0] 0 OpenSearch Security modules loaded so far: []
[2022-08-12T19:14:24,864][ERROR][o.o.s.c.ConfigurationRepository] [opensearch-cluster-data-0] Cannot apply default config (this is maybe not an error!)
org.opensearch.discovery.ClusterManagerNotDiscoveredException: null
	at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAction.java:305) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.2.0.jar:2.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T19:14:24,889][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-data-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T19:14:37,906][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-data-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T19:14:50,909][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-data-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]

ddodoo · August 12, 2022, 10:44pm

@pablo

opensearch-cluster-client-0

Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-2.2.0
Detected OpenSearch Security Version: 2.2.0.0
tee: /usr/share/opensearch/config/opensearch.yml: Permission denied

Enabling OpenSearch Security Plugin
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.2.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.2.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2022-08-12T19:13:19,532][INFO ][o.o.n.Node               ] [opensearch-cluster-client-0] version[2.2.0], pid[45], build[tar/b1017fa3b9a1c781d4f34ecee411e0cdf930a515/2022-08-09T02:27:25.256769336Z], OS[Linux/5.4.0-1089-azure/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.4/17.0.4+8]
[2022-08-12T19:13:19,538][INFO ][o.o.n.Node               ] [opensearch-cluster-client-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK [true]
[2022-08-12T19:13:19,538][INFO ][o.o.n.Node               ] [opensearch-cluster-client-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-8961332606370530363, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx512M, -Xms512M, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2022-08-12T19:13:23,844][WARN ][stderr                   ] [opensearch-cluster-client-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2022-08-12T19:13:23,844][WARN ][stderr                   ] [opensearch-cluster-client-0] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2022-08-12T19:13:23,845][WARN ][stderr                   ] [opensearch-cluster-client-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2022-08-12T19:13:23,929][INFO ][o.o.s.s.t.SSLConfig      ] [opensearch-cluster-client-0] SSL dual mode is disabled
[2022-08-12T19:13:23,930][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] OpenSearch Config path is /usr/share/opensearch/config
[2022-08-12T19:13:24,746][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-client-0] JVM supports TLSv1.3
[2022-08-12T19:13:24,748][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-client-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2022-08-12T19:13:26,031][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-client-0] TLS Transport Client Provider : JDK
[2022-08-12T19:13:26,031][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-client-0] TLS Transport Server Provider : JDK
[2022-08-12T19:13:26,031][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-client-0] TLS HTTP Provider             : JDK
[2022-08-12T19:13:26,031][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-client-0] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2022-08-12T19:13:26,031][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-client-0] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2]
[2022-08-12T19:13:26,756][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] Clustername: opensearch-cluster
[2022-08-12T19:13:26,763][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[2022-08-12T19:13:26,763][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[2022-08-12T19:13:26,763][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] File /usr/share/opensearch/config/kirk.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:26,763][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:26,763][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] File /usr/share/opensearch/config/esnode-key.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:26,764][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] File /usr/share/opensearch/config/esnode.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:26,764][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] File /usr/share/opensearch/config/kirk-key.pem has insecure file permissions (should be 0600)
[2022-08-12T19:13:28,017][INFO ][o.o.p.c.PluginSettings   ] [opensearch-cluster-client-0] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2022-08-12T19:13:29,219][INFO ][o.o.i.r.ReindexPlugin    ] [opensearch-cluster-client-0] ReindexPlugin reloadSPI called
[2022-08-12T19:13:29,226][INFO ][o.o.i.r.ReindexPlugin    ] [opensearch-cluster-client-0] Unable to find any implementation for RemoteReindexExtension
[2022-08-12T19:13:29,336][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-client-0] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2022-08-12T19:13:29,431][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-client-0] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2022-08-12T19:13:29,433][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-client-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-08-12T19:13:29,440][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [aggs-matrix-stats]
[2022-08-12T19:13:29,440][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [analysis-common]
[2022-08-12T19:13:29,440][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [geo]
[2022-08-12T19:13:29,440][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [ingest-common]
[2022-08-12T19:13:29,441][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [ingest-geoip]
[2022-08-12T19:13:29,441][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [ingest-user-agent]
[2022-08-12T19:13:29,441][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [lang-expression]
[2022-08-12T19:13:29,441][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [lang-mustache]
[2022-08-12T19:13:29,441][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [lang-painless]
[2022-08-12T19:13:29,441][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [mapper-extras]
[2022-08-12T19:13:29,441][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [opensearch-dashboards]
[2022-08-12T19:13:29,441][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [parent-join]
[2022-08-12T19:13:29,441][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [percolator]
[2022-08-12T19:13:29,441][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [rank-eval]
[2022-08-12T19:13:29,441][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [reindex]
[2022-08-12T19:13:29,441][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [repository-url]
[2022-08-12T19:13:29,441][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [systemd]
[2022-08-12T19:13:29,441][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded module [transport-netty4]
[2022-08-12T19:13:29,442][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-alerting]
[2022-08-12T19:13:29,442][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-anomaly-detection]
[2022-08-12T19:13:29,442][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-asynchronous-search]
[2022-08-12T19:13:29,442][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-cross-cluster-replication]
[2022-08-12T19:13:29,442][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-geospatial]
[2022-08-12T19:13:29,442][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-index-management]
[2022-08-12T19:13:29,442][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-job-scheduler]
[2022-08-12T19:13:29,442][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-knn]
[2022-08-12T19:13:29,443][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-ml]
[2022-08-12T19:13:29,443][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-notifications]
[2022-08-12T19:13:29,443][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-notifications-core]
[2022-08-12T19:13:29,443][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-observability]
[2022-08-12T19:13:29,443][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-performance-analyzer]
[2022-08-12T19:13:29,443][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-reports-scheduler]
[2022-08-12T19:13:29,443][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-security]
[2022-08-12T19:13:29,443][INFO ][o.o.p.PluginsService     ] [opensearch-cluster-client-0] loaded plugin [opensearch-sql]
[2022-08-12T19:13:29,527][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2022-08-12T19:13:29,558][INFO ][o.o.e.NodeEnvironment    ] [opensearch-cluster-client-0] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/sda1)]], net usable_space [22.9gb], net total_space [28.8gb], types [ext4]
[2022-08-12T19:13:29,559][INFO ][o.o.e.NodeEnvironment    ] [opensearch-cluster-client-0] heap size [512mb], compressed ordinary object pointers [true]
[2022-08-12T19:13:29,660][INFO ][o.o.n.Node               ] [opensearch-cluster-client-0] node name [opensearch-cluster-client-0], node ID [aQ165eaKQ4iFBoQS7gwvAQ], cluster name [opensearch-cluster], roles [map[master:false], map[data:false], map[ingest:false], map[remote_cluster_client:false]]
[2022-08-12T19:13:36,229][WARN ][o.o.s.c.Salt             ] [opensearch-cluster-client-0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2022-08-12T19:13:36,325][INFO ][o.o.s.a.i.AuditLogImpl   ] [opensearch-cluster-client-0] Message routing enabled: true
[2022-08-12T19:13:36,367][INFO ][o.o.s.f.SecurityFilter   ] [opensearch-cluster-client-0] <NONE> indices are made immutable.
[2022-08-12T19:13:37,089][INFO ][o.o.a.b.ADCircuitBreakerService] [opensearch-cluster-client-0] Registered memory breaker.
[2022-08-12T19:13:37,754][INFO ][o.o.m.c.b.MLCircuitBreakerService] [opensearch-cluster-client-0] Registered ML memory breaker.
[2022-08-12T19:13:38,299][INFO ][o.o.t.NettyAllocator     ] [opensearch-cluster-client-0] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
[2022-08-12T19:13:38,420][INFO ][o.o.d.DiscoveryModule    ] [opensearch-cluster-client-0] using discovery type [zen] and seed hosts providers [settings]
[2022-08-12T19:13:39,033][WARN ][o.o.g.DanglingIndicesState] [opensearch-cluster-client-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-08-12T19:13:39,607][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-cluster-client-0] PerformanceAnalyzer Enabled: false
[2022-08-12T19:13:39,648][INFO ][o.o.n.Node               ] [opensearch-cluster-client-0] initialized
[2022-08-12T19:13:39,648][INFO ][o.o.n.Node               ] [opensearch-cluster-client-0] starting ...
[2022-08-12T19:13:39,800][INFO ][o.o.t.TransportService   ] [opensearch-cluster-client-0] publish_address {172.17.0.5:9300}, bound_addresses {0.0.0.0:9300}
[2022-08-12T19:13:40,052][INFO ][o.o.b.BootstrapChecks    ] [opensearch-cluster-client-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-08-12T19:14:10,089][WARN ][o.o.n.Node               ] [opensearch-cluster-client-0] timed out while waiting for initial discovery state - timeout: 30s
[2022-08-12T19:14:10,122][INFO ][o.o.h.AbstractHttpServerTransport] [opensearch-cluster-client-0] publish_address {172.17.0.5:9200}, bound_addresses {0.0.0.0:9200}
[2022-08-12T19:14:10,123][INFO ][o.o.n.Node               ] [opensearch-cluster-client-0] started
[2022-08-12T19:14:10,124][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] Node started
[2022-08-12T19:14:10,124][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-client-0] Will attempt to create index .opendistro_security and default configs if they are absent
[2022-08-12T19:14:10,128][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-client-0] 0 OpenSearch Security modules loaded so far: []
[2022-08-12T19:14:10,130][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-client-0] Background init thread started. Install default config?: true
[2022-08-12T19:14:40,143][ERROR][o.o.s.c.ConfigurationRepository] [opensearch-cluster-client-0] Cannot apply default config (this is maybe not an error!)
org.opensearch.discovery.ClusterManagerNotDiscoveredException: null
	at org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction$AsyncSingleAction$2.onTimeout(TransportClusterManagerNodeAction.java:305) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:394) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:294) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:707) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.2.0.jar:2.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T19:14:40,160][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-client-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-12T19:14:53,179][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-client-0] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
org.opensearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:204) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:190) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:81) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.get.TransportMultiGetAction.doExecute(TransportMultiGetAction.java:58) ~[opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:232) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78) [opensearch-performance-analyzer-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:423) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.client.support.AbstractClient.multiGet(AbstractClient.java:539) [opensearch-2.2.0.jar:2.2.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.loadAsync(ConfigurationLoaderSecurity7.java:207) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationLoaderSecurity7.load(ConfigurationLoaderSecurity7.java:98) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:372) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:318) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:303) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:163) [opensearch-security-2.2.0.0.jar:2.2.0.0]
	at java.lang.Thread.run(Thread.java:833) [?:?]

Topic		Replies	Views
Opensearch security not initialized; Securityadmin.sh not running Security troubleshoot , configure	10	4313	June 20, 2022
OpenSearch Security not initialized Security releases	7	20206	August 8, 2022
Securityadmin.sh no permissions Security	10	894	July 12, 2023
Securityadmin.sh ERR....is not an admin user Security troubleshoot , configure , security-issue	4	741	March 28, 2023
Error securityadmin.sh execution Security	2	27	July 23, 2024

Open Search Security Not Initialized

Related Topics