Errors at opensearch startup

Miky · November 2, 2023, 10:41pm

Opensearch: 2.10

Hello,
When I start opensearch I have these errors
[2023-11-02T23:23:57,435][INFO ][o.o.s.l.LogTypeService ] [node1] Loading builtin types!
[2023-11-02T23:23:57,498][INFO ][o.o.s.l.LogTypeService ] [node1] Indexing [418] fieldMappingDocs from logTypes: 23
[2023-11-02T23:23:57,660][INFO ][o.o.s.l.LogTypeService ] [node1] Loading builtin types!
[2023-11-02T23:23:57,661][INFO ][o.o.s.l.LogTypeService ] [node1] Indexing [418] fieldMappingDocs from logTypes: 23
[2023-11-02T23:23:57,739][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@3c3547a9] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-02T23:23:57,740][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@3c3547a9] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-02T23:23:57,740][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@3c3547a9] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-02T23:23:57,741][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@3c3547a9] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-02T23:23:57,741][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@3c3547a9] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-02T23:23:57,742][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@3c3547a9] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-02T23:23:57,742][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@3c3547a9] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-02T23:23:57,742][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@3c3547a9] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-02T23:23:57,743][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@3c3547a9] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-02T23:23:57,744][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@3c3547a9] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)

I don’t understand where the problem comes from

curl -X GET -u admin:admin “https://localhost:9200/.opendistro_security?pretty”

{
“.opendistro_security” : {
“aliases” : { },
“mappings” : {
“properties” : {
“actiongroups” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“audit” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“config” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“internalusers” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“nodesdn” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“roles” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“rolesmapping” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“tenants” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
},
“whitelist” : {
“type” : “text”,
“fields” : {
“keyword” : {
“type” : “keyword”,
“ignore_above” : 256
}
}
}
}
},
“settings” : {
“index” : {
“replication” : {
“type” : “DOCUMENT”
},
“number_of_shards” : “1”,
“auto_expand_replicas” : “0-all”,
“provided_name” : “.opendistro_security”,
“creation_date” : “1698884762924”,
“number_of_replicas” : “0”,
“uuid” : “hEw67cl8S1mdDu0eku04hg”,
“version” : {
“created” : “136317827”
}
}
}
}
}

curl -X GET -u admin:admin “localhost:9200/_cat/shards?v”

index shard prirep state docs store ip node
.plugins-ml-config 0 p STARTED 1 3.9kb 127.0.0.1 node1
.opensearch-observability 0 p STARTED 0 208b 127.0.0.1 node1
.opensearch-sap-log-types-config 0 p STARTED 127.0.0.1 node1
myindex-000001 0 p STARTED 1 4.2kb 127.0.0.1 node1
.opendistro_security 0 p STARTED 9 31.8kb 127.0.0.1 node1
.kibana_1 0 p STARTED 0 208b 127.0.0.1 node1

curl -X GET -u admin:admin “https://localhost:9200/_cluster/health?pretty”

{
“cluster_name” : “localhost”,
“status” : “green”,
“timed_out” : false,
“number_of_nodes” : 1,
“number_of_data_nodes” : 1,
“discovered_master” : true,
“discovered_cluster_manager” : true,
“active_primary_shards” : 6,
“active_shards” : 6,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 0,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 100.0
}

Thank you

pablo · November 3, 2023, 12:47am

@Miky How do you deploy your cluster?
Do those ERRORs stop after a while?

How long did you wait for a cluster to stabilize?
Do you see a message in the logs “Node <node_name> initialized”?

Please share your config.yml and opensearch.yml files.

Miky · November 3, 2023, 1:25am

Hello Pablo,

I see these errors during the startup but it stops, I see only once.
I see them in /var/log/srv.mydomain.org.log
I start the service at 02:11:12, the errors are displayed at 02:11:39 and it ends writing many logs at 02:11:53, then it wrote a few logs during 4 minutes.

$ cat config.yml
_meta:
type: “config”
config_version: 2

config:
dynamic:
# Set filtered_alias_mode to ‘disallow’ to forbid more than 2 filtered aliases per index
# Set filtered_alias_mode to ‘warn’ to allow more than 2 filtered aliases per index but warns about it (default)
# Set filtered_alias_mode to ‘nowarn’ to allow more than 2 filtered aliases per index silently
#filtered_alias_mode: warn
#do_not_fail_on_forbidden: false
#kibana:
# Kibana multitenancy
#multitenancy_enabled: true
#server_username: kibanaserver
#index: ‘.kibana’
http:
anonymous_auth_enabled: false
xff:
enabled: false
#internalProxies: ‘192.168.0.10|192.168.0.11’ # regex pattern
internalProxies: ‘.*’ # trust all internal proxies, regex pattern
authc:
basic_internal_auth_domain:
description: “Authenticate via HTTP Basic against internal users database”
http_enabled: true
transport_enabled: true
order: 4
http_authenticator:
type: basic
challenge: true
authentication_backend:
type: intern

$ cat
cluster.name: srv. domain. org
node.name: node1
node.max_local_storage_nodes: 2
path.data: /var/opensearch/
path.logs: /var/log/opensearch/
network.host: local
http.port: 9200
discovery.type: single-node
gateway.auto_import_dangling_indices: true
http.compression: false

plugins:
security:
compliance.salt: Salt
ssl:
transport:
enabled: true
pemcert_filepath: cert.crt
pemkey_filepath: key.key.pkcs
pemtrustedcas_filepath: CA.crt
enforce_hostname_verification: false
http:
enabled: true
pemcert_filepath: cert.crt
pemkey_filepath: key.key.pkcs
pemtrustedcas_filepath: CA.crt

plugins.security.authcz.admin_dn:

EMAILADDRESS=mail@domain.org,CN=srv.domain.org,OU=Corp1,O=Corp,L=City,ST=State,C=CR

plugins.security.roles_mapping_resolution: MAPPING_ONLY

plugins.query.datasources.encryption.masterkey: “XXXXXXXXXX”

pablo · November 3, 2023, 12:55pm

@Miky In that case this is expected. The security plugin causes the errors. The plugin is initiated after the OpenSearch service is started. It needs around a minute or less to fully start and stabilize.
You can ignore these errors.

It would be a different story if these errors would continue infinitely. Then it could mean that your security config files are incorrect.

However, according to your previous outputs, you can successfully connect to the cluster using credentials. This means that the security plugin has been successfully initiated.

Regarding your APIs.

curl -X GET -u admin:admin “https://localhost:9200/.opendistro_security?pretty”

To see the content of the .opendistro_security index you must use admin certificates defined in the admin_dn of the opensearch.yml.
Alternatively, you can use the following API calls to check the content of the security configuration.

i.e.

Security config
curl --insecure -X GET -u admin:admin "https://localhost:9200/_plugins/_security/api/securityconfig?pretty"

Internal users
curl --insecure -X GET -u admin:admin "https://localhost:9200/_plugins/_security/api/internalusers?pretty"

Miky · November 3, 2023, 5:48pm

Hello Pablo and thank you for your answers,

Yes indeed I’m able to retrieve the .opendistro_security index

curl -X GET -u admin:admin “https://localhost:9200/.opendistro_security?pretty”

{
  ".opendistro_security" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "actiongroups" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "audit" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "config" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "internalusers" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "nodesdn" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "roles" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "rolesmapping" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "tenants" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "whitelist" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "replication" : {
          "type" : "DOCUMENT"
        },
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-all",
        "provided_name" : ".opendistro_security",
        "creation_date" : "1698884762924",
        "number_of_replicas" : "0",
        "uuid" : "hEw67cl8S1mdDu0eku04hg",
        "version" : {
          "created" : "136317827"
        }
      }
    }
  }
}

Security config
curl --insecure -X GET -u admin:admin "https://localhost:9200/_plugins/_security/api/securityconfig?pretty"
{
  "status" : "FORBIDDEN",
  "message" : "No permission to access REST API: Role based access not enabled.. No client TLS certificate found in request"
}

I’ll solve the REST API problems.

Internal users
curl --insecure -X GET -u admin:admin "https://localhost:9200/_plugins/_security/api/internalusers?pretty"
{
  "status" : "FORBIDDEN",
  "message" : "No permission to access REST API: Role based access not enabled.. No client TLS certificate found in request"
}

pablo · November 3, 2023, 5:54pm

@Miky Check your opensearch.yml file. You should have the following line in that file.

plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

Any user with the listed roles will have access to API calls.

This API will show you just the structure of the index.

curl -X GET -u admin:admin “https://localhost:9200/.opendistro_security?pretty”

To see the content of the index you need to use _search.

curl -X GET --cert <admin.pem> --key <admin.key> “https://localhost:9200/.opendistro_security/_search?pretty”

Miky · November 3, 2023, 6:21pm

Thank you for this plugin command, I’m able to execute the previous queries

curl --insecure -X GET -u admin:admin "https://localhost:9200/_plugins/_security/api/internalusers?pretty"
{
  "logstash" : {
    "hash" : "",
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [
      "logstash"
    ],
    "attributes" : { },
    "description" : "Demo logstash user, using external role mapping",
    "opendistro_security_roles" : [ ],
    "static" : false
  },
  "admin" : {
    "hash" : "",
    "reserved" : true,
    "hidden" : false,
    "backend_roles" : [
      "superuser"
    ],
    "attributes" : { },
    "description" : "Demo admin user",
    "opendistro_security_roles" : [ ],
    "static" : false
  },
  "kibanaserver" : {
    "hash" : "",
    "reserved" : true,
    "hidden" : false,
    "backend_roles" : [ ],
    "attributes" : { },
    "description" : "Demo OpenSearch Dashboards user",
    "opendistro_security_roles" : [ ],
    "static" : false
  },
  "syslogng" : {
    "hash" : "",
    "reserved" : false,
    "hidden" : false,
    "backend_roles" : [
      "logstash"
    ],
    "attributes" : { },
    "description" : "Utilisateur Syslog NG permettant l'ecriture des Syslog dans Opensearch",
    "opendistro_security_roles" : [ ],
    "static" : false
  }
}

curl --insecure -X GET -u admin:admin "https://localhost:9200/_plugins/_security/api/securityconfig?pretty"
{
  "config" : {
    "dynamic" : {
      "filtered_alias_mode" : "warn",
      "disable_rest_auth" : false,
      "disable_intertransport_auth" : false,
      "respect_request_indices_options" : false,
      "kibana" : {
        "multitenancy_enabled" : true,
        "private_tenant_enabled" : true,
        "default_tenant" : "",
        "server_username" : "kibanaserver",
        "index" : ".kibana"
      },
      "http" : {
        "anonymous_auth_enabled" : false,
        "xff" : {
          "enabled" : false,
          "internalProxies" : ".*",
          "remoteIpHeader" : "X-Forwarded-For"
        }
      },
      "authc" : {
        "basic_internal_auth_domain" : {
          "http_enabled" : true,
          "transport_enabled" : true,
          "order" : 4,
          "http_authenticator" : {
            "challenge" : true,
            "type" : "basic",
            "config" : { }
          },
          "authentication_backend" : {
            "type" : "intern",
            "config" : { }
          },
          "description" : "Authenticate via HTTP Basic against internal users database"
        }
      },
      "authz" : { },
      "auth_failure_listeners" : { },
      "do_not_fail_on_forbidden" : false,
      "multi_rolespan_enabled" : true,
      "hosts_resolver_mode" : "ip-only",
      "do_not_fail_on_forbidden_empty" : false
    }
  }
}

Topic		Replies	Views
Configuration Recommendations for OpenSearch and Logstash Integration OpenSearch discuss , troubleshoot , configure	1	54	November 4, 2024
Opensearch-dashboards errors OpenSearch troubleshoot , configure	7	1273	July 21, 2022
"type":"index_not_found_exception","reason":"no such index [.kibana]" OpenSearch Dashboards	0	89	November 14, 2024
Initial Setup Security Plugin Error OpenSearch Dashboards troubleshoot	14	4098	March 21, 2024
Opensearch will not start OpenSearch	10	2261	December 26, 2023

Errors at opensearch startup

curl -X GET -u admin:admin “https://localhost:9200/.opendistro_security?pretty”

curl -X GET -u admin:admin “localhost:9200/_cat/shards?v”

curl -X GET -u admin:admin “https://localhost:9200/_cluster/health?pretty”

Related topics