Opensearch startup errors that disappear over time

OpenSearch
,
1

Good day ! I have a problem with my OpenSearch setup using Docker Compose. I have the following configuration in my `docker-compose.yml` file:

services:

  opensearch:

    image: opensearch:3.7.0

    container_name: opensearch-node

    network_mode: host

    environment:

      - cluster.name=opensearch-cluster

      - discovery.type=single-node

      - plugins.security.dfm_empty_overrides_all=true

      - bootstrap.memory_lock=true

      - "OPENSEARCH_JAVA_OPTS=-Xms10G -Xmx10G"

      - DISABLE_INSTALL_DEMO_CONFIG=true

      - opensearch_performance_analyzer_enabled=false

\#      - "OPENSEARCH_INITIAL_ADMIN_PASSWORD="

      - plugins.security.ssl.transport.pemcert_filepath=/usr/share/opensearch/certs/node.pem

      - plugins.security.ssl.transport.pemkey_filepath=/usr/share/opensearch/certs/node-key.pem

      - plugins.security.ssl.transport.pemtrustedcas_filepath=/usr/share/opensearch/certs/root-ca.pem

      - transport.ssl.enforce_hostname_verification=false

      - plugins.security.ssl.http.enabled=true

      - plugins.security.ssl.http.pemcert_filepath=/usr/share/opensearch/certs/node.pem

      - plugins.security.ssl.http.pemkey_filepath=/usr/share/opensearch/certs/node-key.pem

      - plugins.security.ssl.http.pemtrustedcas_filepath=/usr/share/opensearch/certs/root-ca.pem

      - plugins.security.restapi.roles_enabled=\["all_access", "security_rest_api_access"\]

      - plugins.security.authcz.admin_dn=\["CN=admin-opensearch.local"\]

    env_file:

      - /opt/opensearch/.ldap.env

    ulimits:

      memlock:

        soft: -1

        hard: -1

      nofile:

        soft: 65536

        hard: 65536

    volumes:

      - /data:/usr/share/opensearch/data

      - ./certs/node:/usr/share/opensearch/certs

      - ./security-config:/usr/share/opensearch/config/opensearch-security:ro

    restart: unless-stopped

  opensearch-dashboards:

    image: opensearch-dashboards:3.7.0

    container_name: opensearch-dashboards

    network_mode: host

    environment:

      - DISABLE_SECURITY_DASHBOARDS_PLUGIN=false

    volumes:

      - ./certs/node:/usr/share/opensearch/certs:ro

      - ./opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml:ro

    restart: unless-stopped

Errors:

\[2026-07-17T16:41:19,943\]\[ERROR\]\[o.o.s.a.s.SinkProvider   \] \[1-opensearch01\] Default endpoint could not be created, auditlog will not work properly.

\[2026-07-17T16:41:27,288\]\[WARN \]\[o.o.s.OpenSearchSecurityPlugin\] \[1-opensearch01\] Failed to load API tokens on node start

OpenSearchSecurityException\[Received error while reloading API tokens metadata from index\]; nested: ClusterBlockException\[blocked by: \[SERVICE_UNAVAILABLE/1/state not recovered / initialized\];\];

        at org.opensearch.security.action.apitokens.ApiTokenRepository$1.onFailure(ApiTokenRepository.java:101)

        at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)

        at org.opensearch.action.support.TransportAction$1.onFailure(TransportAction.java:124)

        at org.opensearch.ubi.UbiActionFilter$1.onFailure(UbiActionFilter.java:118)

        at org.opensearch.action.search.TransportSearchAction.lambda$doExecute$2(TransportSearchAction.java:349)

        at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)

        at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)

        at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)

        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:84)

        at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:140)

        at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:105)

        at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$0(TransportSearchAction.java:508)

        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)

        at org.opensearch.search.pipeline.PipelinedRequest.transformRequest(PipelinedRequest.java:73)

        at org.opensearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:515)

        at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:352)

        at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:142)

        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:220)

        at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:41)

        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)

        at org.opensearch.ubi.UbiActionFilter.apply(UbiActionFilter.java:93)

        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)

        at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:120)

        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)

        at org.opensearch.neuralsearch.search.HybridQuerySearchRequestFilter.apply(HybridQuerySearchRequestFilter.java:93)

        at org.opensearch.action.support.ActionFilter.apply(ActionFilter.java:67)

        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)

        at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:81)

        at org.opensearch.action.support.ActionFilter.apply(ActionFilter.java:67)

        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)

        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:313)

        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:186)

        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)

        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190)

        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:109)

        at org.opensearch.transport.client.node.NodeClient.executeLocally(NodeClient.java:113)

        at org.opensearch.transport.client.node.NodeClient.doExecute(NodeClient.java:100)

        at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501)

        at org.opensearch.transport.client.support.AbstractClient.search(AbstractClient.java:632)

        at org.opensearch.security.action.apitokens.ApiTokenIndexHandler.getTokenMetadatas(ApiTokenIndexHandler.java:104)

        at org.opensearch.security.action.apitokens.ApiTokenRepository.reloadApiTokensFromIndex(ApiTokenRepository.java:78)

        at org.opensearch.security.OpenSearchSecurityPlugin.onNodeStarted(OpenSearchSecurityPlugin.java:2396)

        at org.opensearch.node.Node.lambda$start$2(Node.java:2120)

        at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)

        at org.opensearch.node.Node.start(Node.java:2120)

        at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:349)

        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:423)

        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168)

        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159)

        at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110)

        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)

        at org.opensearch.cli.Command.main(Command.java:101)

        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125)

        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91)

Caused by: ClusterBlockException\[blocked by: \[SERVICE_UNAVAILABLE/1/state not recovered / initialized\];\]

        at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:215)

        at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:201)

        at org.opensearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1073)

        at org.opensearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:903)

        at org.opensearch.action.search.TransportSearchAction.lambda$buildRewriteListener$0(TransportSearchAction.java:539)

        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)

        ... 45 more

\[2026-07-17T16:41:27,733\]\[ERROR\]\[o.o.i.i.ManagedIndexCoordinator\] \[1-opensearch01\] get managed-index failed: NoShardAvailableActionException\[No shard available for \[org.opensearch.action.get.MultiGetShardRequest@53c92202\]\]

\[2026-07-17T16:41:27,757\]\[WARN \]\[o.o.s.SecurityAnalyticsPlugin\] \[1-opensearch01\] Failed to initialize LogType config index and builtin log types

\[2026-07-17T16:41:27,757\]\[ERROR\]\[o.o.i.i.ManagedIndexCoordinator\] \[1-opensearch01\] Failed to get ISM policies with templates: Failed to execute phase \[query\], all shards failed

\[2026-07-17T16:41:27,879\]\[ERROR\]\[o.o.s.a.BackendRegistry  \] \[1-opensearch01\] OpenSearch Security not initialized. (you may need to run securityadmin)

These errors only appear during the first 2-3 minutes of startup. After that, they disappear. LDAP authentication and search work correctly. The cluster status is green. I am using a non-replica installation. Increasing the resources did not solve the problem. I believe this is because some operations start before the application is fully initialized. Is there a way to fix these errors?

2

@Ruslan1 Did you miss a dot?

- ./data:/usr/share/opensearch/data
3

@Ruslan1 The error appears because the Security Plugin is initialized later.

OpenSearch Security not initialized. (you may need to run securityadmin)

Regarding the Failed to load API tokens ..., there was a bug reported 2 weeks ago.

How did you initlized the security plugin? Did you run securityadmin.sh?
You could consider using the option below in opensearch.yml

plugins.security.allow_default_init_securityindex: true
4

No, everything is correct. I’m specifying the path to a specific directory

5

@Ruslan1 What permissions are on that directory? Does the /data belong to root?

6

Thanks for the reply. If I understand correctly, setting plugins.security.allow_default_init_securityindex: true will create the index on its own, replacing the current one. I created the configurations in the following folder:
- ./security-config:/usr/share/opensearch/config/opensearch-security:ro
and used the following command:
docker exec opensearch-node /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cd /usr/share/opensearch/config/opensearch-security/ -cacert /usr/share/opensearch/certs/root-ca.pem -cert /usr/share/opensearch/certs/admin.pem -key /usr/share/opensearch/certs/admin-key.pem -icl -nhnv -rev
After that, the index have been created. And when I restart the container I get the errors above.
The /data directory belongs to the user with UUID:GUID 1000:1000 similar to the container

7

@Ruslan1 plugins.security.allow_default_init_securityindex: true doesn’t replace the security index; it only creates it if doesn’t exist.

The index will be created using files from the /usr/share/opensearch/config/opensearch-security folder.
After the index is created, you must use securityadmin.sh, the OpenSearch Dashboards UI, or the APIs to modify the security configuration.

The ‘API tokens …’ error may appear after a restart. However, the strange part is the shards errors.

I used your docker compose and I had no such errors during the initial start-up and restart.

Would you mind sharing the output of the following APIs?

GET _cat/shards
GET _cluster/health
8

Result /_cluster/health

{
  "cluster_name": "opensearch-cluster",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  "discovered_master": true,
  "discovered_cluster_manager": true,
  "active_primary_shards": 112,
  "active_shards": 112,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}

result GET _cat/shards

.ql-datasources                                                                                 0 p STARTED       0    208b 10.111.111.203 1-opensearch01
.opendistro-alerting-alert-history-2026.06.09-1                                                 0 p STARTED                 10.111.111.203 1-opensearch01
security-auditlog-2026.07.10                                                                    0 p STARTED    1094   1.6mb 10.111.111.203 1-opensearch01
.opensearch-sap-windows-findings-2026.07.04-000003                                              0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-reports-instances                                                                   0 p STARTED       0    208b 10.111.111.203 1-opensearch01
.opendistro_security                                                                            0 p STARTED       9  62.5kb 10.111.111.203 1-opensearch01
.opendistro-alerting-config                                                                     0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.06.30-000089                                         0 p STARTED                 10.111.111.203 1-opensearch01
.ds-logs-some-000139                                                             0 p STARTED       0    208b 10.111.111.203 1-opensearch01
.opensearch-sap-windows-alerts-history-2026.06.04-000002                                        0 p STARTED                 10.111.111.203 1-opensearch01
.opensearch-alerting-config-lock                                                                0 p STARTED                 10.111.111.203 1-opensearch01
logstash-2026.06.17                                                                             0 p STARTED       1  26.6kb 10.111.111.203 1-opensearch01
logstash-2026.06.18                                                                             0 p STARTED       1  34.3kb 10.111.111.203 1-opensearch01
logstash-2026.06.15                                                                             0 p STARTED       1  47.4kb 10.111.111.203 1-opensearch01
logstash-2026.06.16                                                                             0 p STARTED       1  27.3kb 10.111.111.203 1-opensearch01
security-auditlog-2026.07.08                                                                    0 p STARTED    7153     4mb 10.111.111.203 1-opensearch01
security-auditlog-2026.07.09                                                                    0 p STARTED     396 796.3kb 10.111.111.203 1-opensearch01
.ds-logs-some-000140                                                             0 p STARTED       0    208b 10.111.111.203 1-opensearch01
.opensearch-sap--job                                                                            0 p STARTED                 10.111.111.203 1-opensearch01
itsec-soc-lists                                                                                 0 p STARTED       5  27.2kb 10.111.111.203 1-opensearch01
.opensearch-sap-linux-alerts-history-2026.06.18-000005                                          0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-alerting-alerts                                                                     0 p STARTED                 10.111.111.203 1-opensearch01
.geospatial-ip2geo-data.geolite2-city.a2db3d7b-25bc-40af-93c2-77bd3c9cec3b                      0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.01-000090                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.06.20-000079                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.20-000105                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.17-000102                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.19-000104                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opensearch-alerting-queries-000001                                                             0 p STARTED                 10.111.111.203 1-opensearch01
top_queries-2026.07.21-44538                                                                    0 p STARTED    4337   4.5mb 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.06.27-000086                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.06.28-000087                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.04-000093                                         0 p STARTED                 10.111.111.203 1-opensearch01
.ds-logs-some-000036                                                             0 p STARTED   50968  15.5mb 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.05-000094                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.06.23-000082                                         0 p STARTED                 10.111.111.203 1-opensearch01
.ds-logs-some-000037                                                             0 p STARTED   51436  15.6mb 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.06.24-000083                                         0 p STARTED                 10.111.111.203 1-opensearch01
.ds-logs-some-000038                                                             0 p STARTED   49175  14.7mb 10.111.111.203 1-opensearch01
.opendistro-alerting-alert-history-2026.07.17-000004                                            0 p STARTED                 10.111.111.203 1-opensearch01
.ds-logs-some-000033                                                             0 p STARTED     388 201.1kb 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.08-000096                                         0 p STARTED                 10.111.111.203 1-opensearch01
.ds-logs-some-000034                                                             0 p STARTED   50821  15.3mb 10.111.111.203 1-opensearch01
.ds-logs-some-000035                                                             0 p STARTED   51549  15.5mb 10.111.111.203 1-opensearch01
.opensearch-sap-linux-detectors-queries-optimized-e0dfbd07-b03b-47bc-9474-15a173a07c07-000001   0 p STARTED                 10.111.111.203 1-opensearch01
itsec-alerts-8.5                                                                                0 p STARTED 1442163   6.6gb 10.111.111.203 1-opensearch01
top_queries-2026.07.22-44539                                                                    0 p STARTED    2580   2.5mb 10.111.111.203 1-opensearch01
logstash-2026.06.19                                                                             0 p STARTED       1  34.4kb 10.111.111.203 1-opensearch01
logstash-2026.06.20                                                                             0 p STARTED       1  46.8kb 10.111.111.203 1-opensearch01
logstash-2026.06.21                                                                             0 p STARTED       1  49.8kb 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.14-000100                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.18-000103                                         0 p STARTED                 10.111.111.203 1-opensearch01
top_queries-2026.07.18-44514                                                                    0 p STARTED    4383   3.9mb 10.111.111.203 1-opensearch01
logstash-2026.06.26                                                                             0 p STARTED       1  49.3kb 10.111.111.203 1-opensearch01
.opendistro-job-scheduler-lock                                                                  0 p STARTED      12  32.4kb 10.111.111.203 1-opensearch01
logstash-2026.06.24                                                                             0 p STARTED       1  34.3kb 10.111.111.203 1-opensearch01
logstash-2026.06.25                                                                             0 p STARTED       1    47kb 10.111.111.203 1-opensearch01
.opensearch-sap-windows-alerts-history-2026.07.04-000003                                        0 p STARTED                 10.111.111.203 1-opensearch01
logstash-2026.06.22                                                                             0 p STARTED       1  29.4kb 10.111.111.203 1-opensearch01
.opensearch-notifications-config                                                                0 p STARTED                 10.111.111.203 1-opensearch01
logstash-2026.06.23                                                                             0 p STARTED       1  29.3kb 10.111.111.203 1-opensearch01
.opensearch-sap-detectors-config                                                                0 p STARTED                 10.111.111.203 1-opensearch01
.opensearch-sap-linux-findings-2026.07.19-000006                                                0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-reports-definitions                                                                 0 p STARTED       0    208b 10.111.111.203 1-opensearch01
.scheduler-geospatial-ip2geo-datasource                                                         0 p STARTED                 10.111.111.203 1-opensearch01
logstash-2026.07.21                                                                             0 p STARTED       1  50.3kb 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.11-000097                                         0 p STARTED                 10.111.111.203 1-opensearch01
logstash-2026.07.22                                                                             0 p STARTED       1  47.2kb 10.111.111.203 1-opensearch01
.opensearch-observability                                                                       0 p STARTED       0    208b 10.111.111.203 1-opensearch01
logstash-2026.07.20                                                                             0 p STARTED       1  29.7kb 10.111.111.203 1-opensearch01
.opensearch-sap-log-types-config                                                                0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.12-000098                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.13-000099                                         0 p STARTED                 10.111.111.203 1-opensearch01
top_queries-2026.07.14-44510                                                                    0 p STARTED    6119   4.4mb 10.111.111.203 1-opensearch01
top_queries-2026.07.17-44513                                                                    0 p STARTED    5192   3.9mb 10.111.111.203 1-opensearch01
.kibana_-152937574_admintenant_1                                                                0 p STARTED       2    10kb 10.111.111.203 1-opensearch01
.opensearch-sap-pre-packaged-rules-config                                                       0 p STARTED                 10.111.111.203 1-opensearch01
top_queries-2026.07.20-44537                                                                    0 p STARTED    4383   4.3mb 10.111.111.203 1-opensearch01
.kibana_1851089145_uganshin_1                                                                   0 p STARTED       1   5.2kb 10.111.111.203 1-opensearch01
.opendistro-ism-config                                                                          0 p STARTED                 10.111.111.203 1-opensearch01
top_queries-2026.07.19-44515                                                                    0 p STARTED    4514     4mb 10.111.111.203 1-opensearch01
.opensearch-sap-windows-detectors-queries-optimized-91f21786-2b08-4ded-b322-3fc0a8c49f52-000001 0 p STARTED                 10.111.111.203 1-opensearch01
logstash-2026.07.18                                                                             0 p STARTED       1  27.7kb 10.111.111.203 1-opensearch01
logstash-2026.07.19                                                                             0 p STARTED       1  34.7kb 10.111.111.203 1-opensearch01
logstash-2026.07.16                                                                             0 p STARTED       1    47kb 10.111.111.203 1-opensearch01
logstash-2026.07.17                                                                             0 p STARTED       1  49.7kb 10.111.111.203 1-opensearch01
logstash-2026.07.14                                                                             0 p STARTED       1  34.3kb 10.111.111.203 1-opensearch01
logstash-2026.07.15                                                                             0 p STARTED       1  29.4kb 10.111.111.203 1-opensearch01
.geospatial-ip2geo-data.geolite2-asn.d95b768e-d9ea-4e94-b099-6b978e925b13                       0 p STARTED                 10.111.111.203 1-opensearch01
logstash-2026.07.13                                                                             0 p STARTED       1  26.6kb 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.06.29-000088                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-alerting-alert-history-2026.07.11-000002                                            0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.06.26-000085                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.06.22-000081                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-alerting-alert-history-2026.07.14-000003                                            0 p STARTED                 10.111.111.203 1-opensearch01
.ds-logs-some-8.5.0-000001                                                         0 p STARTED 7491846   4.4gb 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.03-000092                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.06.25-000084                                         0 p STARTED                 10.111.111.203 1-opensearch01
top_queries-2026.07.16-44512                                                                    0 p STARTED    5938   4.6mb 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.06-000095                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.06.21-000080                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opensearch-sap-linux-alerts-history-2026.07.18-000006                                          0 p STARTED                 10.111.111.203 1-opensearch01
.ds-logs-some-8.5.0-000001                                                                 0 p STARTED  611115 617.4mb 10.111.111.203 1-opensearch01
top_queries-2026.07.15-44511                                                                    0 p STARTED    6235   4.4mb 10.111.111.203 1-opensearch01
.kibana_2                                                                                       0 p STARTED      13  84.6kb 10.111.111.203 1-opensearch01
.kibana_1                                                                                       0 p STARTED      12  89.5kb 10.111.111.203 1-opensearch01
.opensearch-sap-linux-findings-2026.06.19-000005                                                0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-alerting-alert-history-2026.07.19-000005                                            0 p STARTED                 10.111.111.203 1-opensearch01
.plugins-ml-config                                                                              0 p STARTED       1     4kb 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.02-000091                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opendistro-ism-managed-index-history-2026.07.16-000101                                         0 p STARTED                 10.111.111.203 1-opensearch01
.opensearch-sap-windows-findings-2026.06.04-000002                                              0 p STARTED                 10.111.111.203 1-opensearch01

What’s bothering me is that it doesn’t see this index when it starts, but after a while it does. I’m worried it might start creating it during this time. If i use plugins.security.allow_default_init_securityindex: true

9

@Ruslan1 Why do you think it doesn’t see the security index?

Does the security configuration reset whenever you start the cluster?

10

My guess is that this error (\[2026-07-17T16:41:27,879\]\[ERROR\]\[o.o.s.a.BackendRegistry \] \[1-opensearch01\] OpenSearch Security not initialized. (you may need to run securityadmin) ) occurs because the index is not in the started state at the time of startup.And I’m worried that that setting might overdo it.
No, after some time after the launch, the authorization works correctly. I don’t have to do anything extra.

11

I applied the setting. The index was not overwritten. GET /_plugins/_security/api/securityconfig returned the same config as before enabling the setting plugins.security.allow_default_init_securityindex: true Thank you @pablo

12

@pablo Do I need to start separate discussions about the other errors in the current topic, such as:

\[2026-07-17T16:41:27,757\]\[ERROR\]\[o.o.i.i.ManagedIndexCoordinator\] \[1-opensearch01\] Failed to get ISM policies with templates: Failed to execute phase \[query\], all shards failed
13

You’ll see this error in the start-up logs as long as the Security Plugin isn’t fully loaded.
The current docker compose deploys a single node with OpenSearch Dashboards. The error could be caused by OpenSearch Dashboards trying to connect to the OpenSearch node before it is fully initilized.

This behaviour is expected, and you can ignore this error.

14

Let’s keep the conversation here.

Could you check the free space on the hard drive? (df -h)

15 
Filesystem      Size  Used Avail Use% Mounted on
udev             12G     0   12G   0% /dev
tmpfs           2.4G  2.2M  2.4G   1% /run
/dev/sdb1       7.8G  3.2G  4.3G  43% /
tmpfs            12G     0   12G   0% /dev/shm
tmpfs           1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           1.0M     0  1.0M   0% /run/credentials/systemd-resolved.service
/dev/sdc1       988M  1.5M  919M   1% /home
/dev/sdd1       988M  356K  921M   1% /tmp
/dev/sde1        15G  9.3G  4.7G  67% /var
/dev/sdh1        98G   13G   81G  14% /data
/dev/sdf1        15G  2.0G   12G  14% /var/log
/dev/sdg1       988M  276K  921M   1% /var/log/audit
tmpfs           1.0M     0  1.0M   0% /run/credentials/getty@tty1.service
overlay          15G  9.3G  4.7G  67% /var/lib/docker/rootfs/overlayfs/3bdeccb75e5573eefb3c0eac4c0b3e4f0cbe42c40de9423f688d608cdc91fd41
overlay          15G  9.3G  4.7G  67% /var/lib/docker/rootfs/overlayfs/deae073ca52a36006b8ecd89254d67cf31d37f44783ac3a6c92fb48b70a948b2
overlay          15G  9.3G  4.7G  67% /var/lib/docker/rootfs/overlayfs/88fdfd43cd30c88cf098c79b18558132e0cbe43e3a4d77db9d924ea533cb35bc
tmpfs           2.4G  4.0K  2.4G   1% /run/user/1040
16

@Ruslan1 Is this output after starting the cluster? What Linux distro did you use?
Where is the Docker host deployed? (AWS, local machine, Windows Server etc??)

Do you see those shard errors at every start-up or this was a one-time issue?

17

I use Debian 13. Host deployed on server virtualization. I see this problem about once every 3-5 hours

18

Regarding

You should see something like this in the logs after a while.

[2026-07-21T14:12:57,395][INFO ][o.o.s.SecurityAnalyticsPlugin] [opensearch-node1] LogType config index successfully created and builtin log types loaded

This will result with the following index.

How many CPU cores and RAM did you assign to that Debian? Is it on-prem or cloud VM?

19

RAM: 24
CPU: 16
TYPE: Cloud VM
I don’t find .opensearch-sap-log-types-config index

image
image2550×1036 296 KB

20

I didn’t find this line.