Remote state fails to write because of a wrong generated path

OpenSearch
,
1

Versions (OpenSearch): 3.6.0 and 3.7.0

Describe the issue: Enabling remote store (minio) leads to state index metadata failing to get written because the path is incorrectly constructed
Instead of sending the files to:
opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/…

it tries to send the files to:
1eGaf8OIz7c//opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/…

I’ve been struggling with this for a week already, please do not hesitate to ask questions.

Configuration:
security enabled
plugins: prometheus-exporter and repository-s3
brand new cluster (everything removed on the remote storage and in path.data on all nodes) or following the migration guide ( Migrating to remote-backed storage | OpenSearch Documentation ) after setting up a working, empty cluster yields the same result.
The roles were originally [master,data,ingest] for all 5 nodes, and I changed to 3 [master] and 2 [data,ingest] hoping it would help, but no change.

cluster.name: opensearch-poc-001
discovery.seed_hosts: ["node-001.mydomain","node-002.mydomain","node-003.mydomain"]
cluster.initial_cluster_manager_nodes: ["node-001.mydomain","node-002.mydomain","node-003.mydomain"]
s3.client.default.endpoint: http://localhost:9000
s3.client.default.path_style_access: true
cluster.remote_store.state.enabled: true
node.attr.remote_store.segment.repository: repo-segment
node.attr.remote_store.translog.repository: repo-translog
node.attr.remote_store.state.repository: repo-state
node.attr.remote_store.repository.repo-segment.type: s3
node.attr.remote_store.repository.repo-segment.settings.bucket: search-opensearch-remote-store
node.attr.remote_store.repository.repo-segment.settings.base_path: /opensearch-poc-001/segment
node.attr.remote_store.repository.repo-segment.settings.region: unused-by-minio
node.attr.remote_store.repository.repo-translog.type: s3
node.attr.remote_store.repository.repo-translog.settings.bucket: search-opensearch-remote-store
node.attr.remote_store.repository.repo-translog.settings.base_path: /opensearch-poc-001/translog
node.attr.remote_store.repository.repo-translog.settings.region: unused-by-minio
node.attr.remote_store.repository.repo-state.type: s3
node.attr.remote_store.repository.repo-state.settings.bucket: search-opensearch-remote-store
node.attr.remote_store.repository.repo-state.settings.base_path: /opensearch-poc-001/state
node.attr.remote_store.repository.repo-state.settings.region: unused-by-minio
node.name: node-001.mydomain
network.host: 0.0.0.0
http.port: 9200
bootstrap.memory_lock: true
path.data: /data/opensearch
path.logs: /var/log/opensearch/
node.roles: [master]
thread_pool.search.size: 32
plugins.security.allow_default_init_securityindex: true
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.ssl.transport.pemcert_filepath: /home/opensearchuser/usr/opensearch/conf/node.pem
plugins.security.ssl.transport.pemkey_filepath: /home/opensearchuser/usr/opensearch/conf/node.key
plugins.security.ssl.transport.pemtrustedcas_filepath: /home/opensearchuser/usr/opensearch/conf/root.pem
transport.ssl.enforce_hostname_verification : false
transport.ssl.resolve_hostname: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /home/opensearchuser/usr/opensearch/conf/node.pem
plugins.security.ssl.http.pemkey_filepath: /home/opensearchuser/usr/opensearch/conf/node.key
plugins.security.ssl.http.pemtrustedcas_filepath: /home/opensearchuser/usr/opensearch/conf/root.pem
plugins.security.nodes_dn:
- O=mycompany,L=mycity,C=CC,CN=node-001.mydomain
- O=mycompany,L=mycity,C=CC,CN=node-002.mydomain
- O=mycompany,L=mycity,C=CC,CN=node-003.mydomain
- O=mycompany,L=mycity,C=CC,CN=node-004.mydomain
- O=mycompany,L=mycity,C=CC,CN=node-005.mydomain
plugins.security.authcz.admin_dn:
- CN=Admin,O=mycompany,L=mycity,C=CC
prometheus.metric_name.prefix: opensearch_

Relevant Logs or Screenshots:
(Note that the “Object name contains unsupported characters” is very likely minio rejecting the “something//” at the start of the object path. It’s not supposed to be there in the first place, the goal of this ticket is to understand where it comes from)
From the opensearch logs:

(message too long, will post in a reply)

From the minio logs:

   LOG: 07:52:31.853 [200 OK]  http://node-059.myminio:9000 PUT /search-opensearch-remote-store//opensearch-poc-001/segment/remote-index-path/remote_path_kqUjmrS8RdadKocJG7kfuA#4#1#XJ4uc5PQTXOEd3adU45joA 127.0.0.1         9.514ms      ↑ 1.5 KiB ↓ 524 B
   LOG: 07:52:31.858 [200 OK]  http://node-028.myminio:9000 PUT /search-opensearch-remote-store//opensearch-poc-001/translog/remote-index-path/remote_path_kqUjmrS8RdadKocJG7kfuA#4#1#J-7nvXYmQo-ScgmD7GxivA 127.0.0.1         15.07ms      ↑ 1.4 KiB ↓ 524 B
   LOG: 07:53:01.957 [200 OK]  http://node-060.myminio:9000 GET /search-opensearch-remote-store?list-type=2&delimiter=%2F&max-keys=1&prefix=%2Fopensearch-poc-001%2Fstate%2Fb3BlbnNlYXJjaC1wb2MtMDAx%2Fcluster-state%2FtA3hpMv3R_6zIFTb7Aybhg%2Fmanifest%2Fmanifest__ 127.0.0.1         4.957ms      ↑ 717 B ↓ 840 B
   LOG: 07:53:01.965 [200 OK]  http://node-017.myminio:9000 PUT /search-opensearch-remote-store//opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/tA3hpMv3R_6zIFTb7Aybhg/global-metadata/settings__9223372036854775765__9223370249728793849__1 127.0.0.1         6.472ms      ↑ 967 B ↓ 524 B
   LOG: 07:53:01.967 [200 OK]  http://node-040.myminio:9000 PUT /search-opensearch-remote-store//opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/tA3hpMv3R_6zIFTb7Aybhg/global-metadata/templates__9223372036854775765__9223370249728793847__1 127.0.0.1         6.685ms      ↑ 968 B ↓ 524 B
   LOG: 07:53:01.968 [200 OK]  http://node-045.myminio:9000 PUT /search-opensearch-remote-store//opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/tA3hpMv3R_6zIFTb7Aybhg/global-metadata/custom--index-graveyard__9223372036854775765__9223370249728793846__1 127.0.0.1         5.998ms      ↑ 960 B ↓ 524 B
   LOG: 07:53:01.968 [200 OK]  http://node-007.myminio:9000 PUT /search-opensearch-remote-store//opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/tA3hpMv3R_6zIFTb7Aybhg/global-metadata/coordination__9223372036854775765__9223370249728793848__1 127.0.0.1         8.028ms      ↑ 1.2 KiB ↓ 524 B
   LOG: 07:53:01.968 [200 OK]  http://node-021.myminio:9000 PUT /search-opensearch-remote-store//opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/tA3hpMv3R_6zIFTb7Aybhg/global-metadata/custom--repositories__9223372036854775765__9223370249728793847__1 127.0.0.1         7.162ms      ↑ 1.4 KiB ↓ 524 B
   LOG: 07:53:01.976 [200 OK]  http://node-027.myminio:9000 PUT /search-opensearch-remote-store//opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/tA3hpMv3R_6zIFTb7Aybhg/manifest/manifest__9223372036854775786__9223372036854775762__P__9223370249728793838__4 127.0.0.1         6.383ms      ↑ 2.7 KiB ↓ 524 B
   LOG: 07:53:02.035 [200 OK]  http://node-051.myminio:9000 PUT /search-opensearch-remote-store//opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/tA3hpMv3R_6zIFTb7Aybhg/manifest/manifest__9223372036854775786__9223372036854775762__C__9223370249728793787__4 127.0.0.1         13.336ms     ↑ 2.7 KiB ↓ 524 B
   LOG: 07:53:02.049 [200 OK]  http://node-041.myminio:9000 PUT /search-opensearch-remote-store//opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/tA3hpMv3R_6zIFTb7Aybhg/manifest/manifest__9223372036854775786__9223372036854775761__P__9223370249728793767__4 127.0.0.1         7.215ms      ↑ 2.7 KiB ↓ 524 B
   LOG: 07:53:02.078 [200 OK]  http://node-007.myminio:9000 PUT /search-opensearch-remote-store//opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/tA3hpMv3R_6zIFTb7Aybhg/manifest/manifest__9223372036854775786__9223372036854775761__C__9223370249728793736__4 127.0.0.1         5.472ms      ↑ 2.7 KiB ↓ 524 B
   LOG: 07:53:02.091 [400 Bad Request]  http://node-015.myminio:9000 PUT /search-opensearch-remote-store/6AvifHbkqDA//opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/tA3hpMv3R_6zIFTb7Aybhg/index/iEETRzLGRUS_kws5h47AiQ/iEETRzLGRUS_kws5h47AiQ/metadata__9223372036854775803__9223370249728793722__2 127.0.0.1         3.508ms      ↑ 930 B ↓ 1.2 KiB
2 
[2026-08-19T00:00:00,230][INFO ][o.o.s.c.ConfigurationRepository] [node-001.mydomain] Wait for cluster to be available ...
[2026-08-19T00:00:01,233][INFO ][o.o.s.c.ConfigurationRepository] [node-001.mydomain] Wait for cluster to be available ...
[2026-08-19T00:00:01,404][ERROR][o.o.s.a.BackendRegistry  ] [node-001.mydomain] OpenSearch Security not initialized. Cluster manager not present (you may need to run securityadmin)
[2026-08-19T00:00:01,404][ERROR][o.o.s.a.BackendRegistry  ] [node-001.mydomain] OpenSearch Security not initialized. Cluster manager not present (you may need to run securityadmin)
[2026-08-19T00:00:01,405][ERROR][o.o.s.a.BackendRegistry  ] [node-001.mydomain] OpenSearch Security not initialized. Cluster manager not present (you may need to run securityadmin)
[2026-08-19T00:00:01,405][ERROR][o.o.s.a.BackendRegistry  ] [node-001.mydomain] OpenSearch Security not initialized. Cluster manager not present (you may need to run securityadmin)
[2026-08-19T00:00:02,233][INFO ][o.o.s.c.ConfigurationRepository] [node-001.mydomain] Wait for cluster to be available ...
[2026-08-19T00:00:02,262][INFO ][o.o.c.c.JoinHelper       ] [node-001.mydomain] failed to join {node-002.mydomain}{Na4RtAgKTHqWff5EyMV79A}{dhOhge6lTteMLuPBz8fEQQ}{100.120.245.87}{100.120.245.87:9300}{m}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={node-001.mydomain}{kZl6m0yyTsePoY2BUiF8qw}{dVpki-iAR2iCE-h3xXH9fQ}{100.120.245.86}{100.120.245.86:9300}{m}{shard_indexing_pressure_enabled=true}, minimumTerm=2136, optionalJoin=Optional[Join{term=2137, lastAcceptedTerm=4, lastAcceptedVersion=53, sourceNode={node-001.mydomain}{kZl6m0yyTsePoY2BUiF8qw}{dVpki-iAR2iCE-h3xXH9fQ}{100.120.245.86}{100.120.245.86:9300}{m}{shard_indexing_pressure_enabled=true}, targetNode={node-002.mydomain}{Na4RtAgKTHqWff5EyMV79A}{dhOhge6lTteMLuPBz8fEQQ}{100.120.245.87}{100.120.245.87:9300}{m}{shard_indexing_pressure_enabled=true}}]}
org.opensearch.transport.RemoteTransportException: [node-002.mydomain][100.120.245.87:9300][internal:cluster/coordination/join]
Caused by: org.opensearch.cluster.coordination.FailedToCommitClusterStateException: publishing failed
        at org.opensearch.cluster.coordination.Coordinator.publish(Coordinator.java:1426)
        at org.opensearch.cluster.service.ClusterManagerService.publish(ClusterManagerService.java:361)
        at org.opensearch.cluster.service.ClusterManagerService.runTasks(ClusterManagerService.java:343)
        at org.opensearch.cluster.service.ClusterManagerService$Batcher.run(ClusterManagerService.java:214)
        at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:206)
        at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:264)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:952)
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:299)
        at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
        at java.lang.Thread.run(Thread.java:1474)
Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: remote_state_transfer_exception: Exception during transfer of following metadata to Remote - settings, coordination, templates, custom--repositories, custom--index-graveyard, .plugins-ml-config, security-auditlog-2026.08.18, .kibana, .ql-datasources, .opensearch-sap-log-types-config, top_queries-2026.08.18-74305, .opendistro_security
        at org.opensearch.gateway.remote.RemoteClusterStateService.writeMetadataInParallel(RemoteClusterStateService.java:860)
        at org.opensearch.gateway.remote.RemoteClusterStateService.writeFullMetadata(RemoteClusterStateService.java:326)
        at org.opensearch.gateway.GatewayMetaState$RemotePersistedState.setLastAcceptedState(GatewayMetaState.java:774)
        at org.opensearch.cluster.coordination.CoordinationState.handlePrePublish(CoordinationState.java:586)
        at org.opensearch.cluster.coordination.Coordinator.publish(Coordinator.java:1421)
        ... 11 more
        Suppressed: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: remote_state_transfer_exception: Upload failed for .opensearch-sap-log-types-config
                at org.opensearch.gateway.remote.RemoteIndexMetadataManager.lambda$getWrappedWriteListener$1(RemoteIndexMetadataManager.java:147)
                at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
                at org.opensearch.repositories.s3.S3BlobContainer.lambda$createFileCompletableFuture$0(S3BlobContainer.java:360)
                at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:884)
                at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:862)
                at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531)
                at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2221)
                at org.opensearch.repositories.s3.async.AsyncTransferManager.lambda$uploadInOneChunk$2(AsyncTransferManager.java:425)
                at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:955)
                at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:932)
                at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531)
                at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2221)
                at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
                at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:884)
                at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:862)
                ... 38 more
        Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: completion_exception: software.amazon.awssdk.services.s3.model.S3Exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD0BCD006837B5, Extended Request ID: 66a10bccb651c11a3fd13e84eaf8b599a0f6a9048624e1b26d0a7e3cd6322d49) (SDK Attempt Count: 1)
                at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:64)
                at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
                at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:955)
                at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:932)
                ... 28 more
        Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: s3_exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD0BCD006837B5, Extended Request ID: 66a10bccb651c11a3fd13e84eaf8b599a0f6a9048624e1b26d0a7e3cd6322d49) (SDK Attempt Count: 1)
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:113)
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:61)
                at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
                ... 21 more
        Suppressed: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: remote_state_transfer_exception: Upload failed for .ql-datasources
                at org.opensearch.gateway.remote.RemoteIndexMetadataManager.lambda$getWrappedWriteListener$1(RemoteIndexMetadataManager.java:147)
                at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
                at org.opensearch.repositories.s3.S3BlobContainer.lambda$createFileCompletableFuture$0(S3BlobContainer.java:360)
                at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:884)
                at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:862)
                at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531)
                at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2221)
                at org.opensearch.repositories.s3.async.AsyncTransferManager.lambda$uploadInOneChunk$2(AsyncTransferManager.java:425)
                at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:955)
                at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:932)
                at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531)
                at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2221)
                at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
                at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:884)
                at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:862)
                ... 38 more
        Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: completion_exception: software.amazon.awssdk.services.s3.model.S3Exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD0BCD00632B55, Extended Request ID: d4e2dc387b4f31176a3e238e7b99dbff1fc765714317274b83be58feabac44ee) (SDK Attempt Count: 1)
                at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:64)
                at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
                at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:955)
                at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:932)
                ... 28 more
        Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: s3_exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD0BCD00632B55, Extended Request ID: d4e2dc387b4f31176a3e238e7b99dbff1fc765714317274b83be58feabac44ee) (SDK Attempt Count: 1)
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:113)
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:61)
                at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
                ... 21 more
        Suppressed: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: remote_state_transfer_exception: Upload failed for .plugins-ml-config
                at org.opensearch.gateway.remote.RemoteIndexMetadataManager.lambda$getWrappedWriteListener$1(RemoteIndexMetadataManager.java:147)
                at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
                at org.opensearch.repositories.s3.S3BlobContainer.lambda$createFileCompletableFuture$0(S3BlobContainer.java:360)
                at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:884)
                at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:862)
                at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531)
                at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2221)
                at org.opensearch.repositories.s3.async.AsyncTransferManager.lambda$uploadInOneChunk$2(AsyncTransferManager.java:425)
                at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:955)
                at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:932)
                at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531)
                at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2221)
                at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
                at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:884)
                at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:862)
                ... 38 more
        Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: completion_exception: software.amazon.awssdk.services.s3.model.S3Exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD0BCD005F1824, Extended Request ID: 26ae47b7a61b78f6e2bff2f4272fa9bc92088140887a43f9e6049016868568f6) (SDK Attempt Count: 1)
                at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:64)
                at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
                at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:955)
                at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:932)
                ... 28 more
        Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: s3_exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD0BCD005F1824, Extended Request ID: 26ae47b7a61b78f6e2bff2f4272fa9bc92088140887a43f9e6049016868568f6) (SDK Attempt Count: 1)
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:113)
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:61)
                at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
                ... 21 more
        Suppressed: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: remote_state_transfer_exception: Upload failed for .opendistro_security
                at org.opensearch.gateway.remote.RemoteIndexMetadataManager.lambda$getWrappedWriteListener$1(RemoteIndexMetadataManager.java:147)
                at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
                at org.opensearch.repositories.s3.S3BlobContainer.lambda$createFileCompletableFuture$0(S3BlobContainer.java:360)
                at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:884)
                at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:862)
                at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531)
                at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2221)
                at org.opensearch.repositories.s3.async.AsyncTransferManager.lambda$uploadInOneChunk$2(AsyncTransferManager.java:425)
                at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:955)
                at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:932)
                at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531)
                at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2221)
                at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
                at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:884)
                at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:862)
                ... 38 more
        Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: completion_exception: software.amazon.awssdk.services.s3.model.S3Exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD0BCD0069EC06, Extended Request ID: a843630abdad9f74e6b2e4ef116ccd20e04fa66ac6f665350522d56c57985da6) (SDK Attempt Count: 1)
                at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:64)
                at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
                at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:955)
                at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:932)
                ... 28 more
        Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: s3_exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD0BCD0069EC06, Extended Request ID: a843630abdad9f74e6b2e4ef116ccd20e04fa66ac6f665350522d56c57985da6) (SDK Attempt Count: 1)
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:113)
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:61)
                at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
                ... 21 more
        Suppressed: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: remote_state_transfer_exception: Upload failed for top_queries-2026.08.18-74305
                at org.opensearch.gateway.remote.RemoteIndexMetadataManager.lambda$getWrappedWriteListener$1(RemoteIndexMetadataManager.java:147)
                at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
                at org.opensearch.repositories.s3.S3BlobContainer.lambda$createFileCompletableFuture$0(S3BlobContainer.java:360)
                at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:884)
                at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:862)
                at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531)
                at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2221)
                at org.opensearch.repositories.s3.async.AsyncTransferManager.lambda$uploadInOneChunk$2(AsyncTransferManager.java:425)
                at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:955)
                at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:932)
                at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531)
                at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2221)
                at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
                at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:884)
                at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:862)
                ... 38 more
        Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: completion_exception: software.amazon.awssdk.services.s3.model.S3Exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD0BCD00675A40, Extended Request ID: 4c2011a5a48423219ab3bfe3ade176f1adee433910745d064007b3d03f57b142) (SDK Attempt Count: 1)
                at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:64)
                at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
                at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:955)
                at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:932)
                ... 28 more
        Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: s3_exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD0BCD00675A40, Extended Request ID: 4c2011a5a48423219ab3bfe3ade176f1adee433910745d064007b3d03f57b142) (SDK Attempt Count: 1)
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:113)
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:61)
                at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
                ... 21 more
        Suppressed: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: remote_state_transfer_exception: Upload failed for .kibana
                at org.opensearch.gateway.remote.RemoteIndexMetadataManager.lambda$getWrappedWriteListener$1(RemoteIndexMetadataManager.java:147)
                at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
                at org.opensearch.repositories.s3.S3BlobContainer.lambda$createFileCompletableFuture$0(S3BlobContainer.java:360)
                at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:884)
                at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:862)
                at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531)
                at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2221)
                at org.opensearch.repositories.s3.async.AsyncTransferManager.lambda$uploadInOneChunk$2(AsyncTransferManager.java:425)
                at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:955)
                at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:932)
                at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531)
                at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2221)
                at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
                at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:884)
                at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:862)
                ... 38 more
        Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: completion_exception: software.amazon.awssdk.services.s3.model.S3Exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD0BCD006761FD, Extended Request ID: e177a3e7503734b7163939eba2cf361a18771f748d842027fd771a450c65a725) (SDK Attempt Count: 1)
                at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:64)
                at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
                at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:955)
                at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:932)
                ... 28 more
        Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: s3_exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD0BCD006761FD, Extended Request ID: e177a3e7503734b7163939eba2cf361a18771f748d842027fd771a450c65a725) (SDK Attempt Count: 1)
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:113)
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:61)
                at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
                ... 21 more
        Suppressed: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: remote_state_transfer_exception: Upload failed for security-auditlog-2026.08.18
                at org.opensearch.gateway.remote.RemoteIndexMetadataManager.lambda$getWrappedWriteListener$1(RemoteIndexMetadataManager.java:147)
                at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
                at org.opensearch.repositories.s3.S3BlobContainer.lambda$createFileCompletableFuture$0(S3BlobContainer.java:360)
                at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:884)
                at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:862)
                at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531)
                at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2221)
                at org.opensearch.repositories.s3.async.AsyncTransferManager.lambda$uploadInOneChunk$2(AsyncTransferManager.java:425)
                at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:955)
                at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:932)
                at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:531)
                at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2221)
                at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
                at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:884)
                at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:862)
                ... 38 more
        Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: completion_exception: software.amazon.awssdk.services.s3.model.S3Exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD0BCD00659905, Extended Request ID: 098e860c6d331155e3a4e37779ef7f222cf6c1ae0fb6fb6d6c77e658c73cbd87) (SDK Attempt Count: 1)
                at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:64)
                at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
                at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:955)
                at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:932)
                ... 28 more
        Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: s3_exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD0BCD00659905, Extended Request ID: 098e860c6d331155e3a4e37779ef7f222cf6c1ae0fb6fb6d6c77e658c73cbd87) (SDK Attempt Count: 1)
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:113)
                at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:61)
                at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
                ... 21 more
[2026-08-19T00:00:03,233][INFO ][o.o.s.c.ConfigurationRepository] [node-001.mydomain] Wait for cluster to be available ...
[2026-08-19T00:00:03,906][ERROR][o.o.s.a.BackendRegistry  ] [node-001.mydomain] OpenSearch Security not initialized. Cluster manager not present (you may need to run securityadmin)
3

Clarification: All these messages are looping (until I kill OpenSearch). Election never completes and indexes like the security one can never be created.

4

Hi @Camusensei, I was able to reproduce your errors.

I’m getting the error below

Exception when listing blobs by prefix [B00100101011011//opensearch-poc-001/segment/2-P25FpGQPy0ANAaLCqJSQ/0/segments/metadata/metadata]
...
[2026-08-19T15:56:08,356][WARN ][o.o.c.r.a.AllocationService] [opensearch-node2] failing shard [failed shard, shard [.plugins-ml-config][0], node[qxp4nazmSlit7jcJepTAOg], [P], recovery_source[new shard recovery], s[INITIALIZING], a[id=B3Oz1LrzQ5qtomOMmZ5UpQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2026-08-19T15:56:08.275Z], failed_attempts[4], failed_nodes[[qxp4nazmSlit7jcJepTAOg, 8Ova86xmRNGMVCHb7TAVgA]], delayed=false, details[failed shard on node [8Ova86xmRNGMVCHb7TAVgA]: failed to create shard, failure IOException[java.io.IOException: Exception when listing blobs by prefix [B00100101011011//opensearch-poc-001/segment/2-P25FpGQPy0ANAaLCqJSQ/0/segments/metadata/metadata]]; nested: IOException[Exception when listing blobs by prefix [B00100101011011//opensearch-poc-001/segment/2-P25FpGQPy0ANAaLCqJSQ/0/segments/metadata/metadata]]; nested: S3Exception[Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD3FF9924BA178, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8) (SDK Attempt Count: 1)]; ], allocation_status[no_attempt]], message [failed to create shard], failure [IOException[java.io.IOException: Exception when listing blobs by prefix [B00100101011011//opensearch-poc-001/segment/2-P25FpGQPy0ANAaLCqJSQ/0/segments/metadata/metadata]]; nested: IOException[Exception when listing blobs by prefix [B00100101011011//opensearch-poc-001/segment/2-P25FpGQPy0ANAaLCqJSQ/0/segments/metadata/metadata]]; nested: NotSerializableExceptionWrapper[s3_exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD3FF996FDCC51, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8) (SDK Attempt Count: 1)]; ], markAsStale [true]]
opensearch-node2_3.6.0       | java.io.IOException: java.io.IOException: Exception when listing blobs by prefix [B00100101011011//opensearch-poc-001/segment/2-P25FpGQPy0ANAaLCqJSQ/0/segments/metadata/metadata]
opensearch-node2_3.6.0       |  at org.opensearch.index.store.RemoteDirectory.listFilesByPrefixInLexicographicOrder(RemoteDirectory.java:155)
opensearch-node2_3.6.0       |  at org.opensearch.index.store.RemoteSegmentStoreDirectory.readLatestMetadataFile(RemoteSegmentStoreDirectory.java:261)
opensearch-node2_3.6.0       |  at org.opensearch.index.store.RemoteSegmentStoreDirectory.init(RemoteSegmentStoreDirectory.java:178)
opensearch-node2_3.6.0       |  at org.opensearch.index.store.RemoteSegmentStoreDirectory.<init>(RemoteSegmentStoreDirectory.java:165)
opensearch-node2_3.6.0       |  at org.opensearch.index.store.RemoteSegmentStoreDirectoryFactory.newDirectory(RemoteSegmentStoreDirectoryFactory.java:176)
opensearch-node2_3.6.0       |  at org.opensearch.index.store.RemoteSegmentStoreDirectoryFactory.newDirectory(RemoteSegmentStoreDirectoryFactory.java:71)
opensearch-node2_3.6.0       |  at org.opensearch.index.IndexService.createShard(IndexService.java:714)
opensearch-node2_3.6.0       |  at org.opensearch.indices.IndicesService.createShard(IndicesService.java:1333)
opensearch-node2_3.6.0       |  at org.opensearch.indices.IndicesService.createShard(IndicesService.java:235)
opensearch-node2_3.6.0       |  at org.opensearch.indices.cluster.IndicesClusterStateService.createShard(IndicesClusterStateService.java:692)
opensearch-node2_3.6.0       |  at org.opensearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:669)
opensearch-node2_3.6.0       |  at org.opensearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:314)
opensearch-node2_3.6.0       |  at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:660)
opensearch-node2_3.6.0       |  at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:646)
opensearch-node2_3.6.0       |  at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:595)
opensearch-node2_3.6.0       |  at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:516)
opensearch-node2_3.6.0       |  at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:212)
opensearch-node2_3.6.0       |  at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:952)
opensearch-node2_3.6.0       |  at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:299)
opensearch-node2_3.6.0       |  at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:262)
opensearch-node2_3.6.0       |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
opensearch-node2_3.6.0       |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
opensearch-node2_3.6.0       |  at java.lang.Thread.run(Thread.java:1474)
opensearch-node2_3.6.0       | Caused by: java.io.IOException: Exception when listing blobs by prefix [B00100101011011//opensearch-poc-001/segment/2-P25FpGQPy0ANAaLCqJSQ/0/segments/metadata/metadata]
opensearch-node2_3.6.0       |  at org.opensearch.repositories.s3.S3BlobContainer.listBlobsByPrefixInSortedOrder(S3BlobContainer.java:476)
opensearch-node2_3.6.0       |  at org.opensearch.common.blobstore.BlobContainer.listBlobsByPrefixInSortedOrder(BlobContainer.java:349)
opensearch-node2_3.6.0       |  at org.opensearch.index.store.RemoteDirectory.listFilesByPrefixInLexicographicOrder(RemoteDirectory.java:144)
opensearch-node2_3.6.0       |  ... 22 more
opensearch-node2_3.6.0       | Caused by: org.opensearch.core.common.io.stream.NotSerializableExceptionWrapper: s3_exception: Object name contains unsupported characters. (Service: S3, Status Code: 400, Request ID: 18CD3FF996FDCC51, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8) (SDK Attempt Count: 1)

OpenSearch is referring to this exact place in the path B00100101011011//opensearch-poc-001

I fixed that by removing the leading / in the base_path

Change your configuration to the one below.

node.attr.remote_store.repository.repo-segment.settings.base_path: opensearch-poc-001/segment
node.attr.remote_store.repository.repo-translog.settings.base_path: opensearch-poc-001/translog
node.attr.remote_store.repository.repo-state.settings.base_path: opensearch-poc-001/state
5

Wow, thank you!!

That was indeed the issue, I didn’t even expect a reply this fast!
Kudos to you good sir!

6

It’s working now so it’s solved as far I’m concerned but I don’t think it’s normal that it’s writing outside the directory it’s supposed to write in. If permissions were enforced on my bucket, this would have sparked a new issue.
Also, this is my first cluster but I plan to have a lot of them, I can’t have another cluster writing to the same bucket if it’s not respecting the location it writes to :cry:

Here are a few files from the listing of my bucket, note the r10110101110000, w00110110111110, y00110100011011, y10111101100000, etc. folders.
(note that I am back to 25 nodes now)

[2026-08-20 03:29:12 UTC]   481B STANDARD opensearch-poc-001/segment/remote-index-path/remote_path_6EN_uPXNR-i4u0TGeblgPA#2#1#WqPcz3HYSeqSdefrh2fNzA
[2026-08-20 03:33:42 UTC]   481B STANDARD opensearch-poc-001/segment/remote-index-path/remote_path_FmfgUgKwRuG9_CxWYXbrkw#4#1#1gJKmojAQ--r7DjtTHKaJw
[2026-08-20 03:29:16 UTC]   481B STANDARD opensearch-poc-001/segment/remote-index-path/remote_path_Hvsn3g4mSDqPgSZQBvRWcw#2#1#7vOux1x8ST-F9dpsWWkWxA
[2026-08-20 03:29:12 UTC]   481B STANDARD opensearch-poc-001/segment/remote-index-path/remote_path_R6jVVl-kQ9ShZdwVVP5a_g#2#1#U7OL_Dc8Qoaf3CVJ39GdUA
[2026-08-20 03:29:16 UTC]   481B STANDARD opensearch-poc-001/segment/remote-index-path/remote_path_YjQUpXO9Sz2Tv1jzaoE9Lw#2#1#WXkKV_9JRZ6lWNMs7DQ7aA
[2026-08-20 03:29:22 UTC]   481B STANDARD opensearch-poc-001/segment/remote-index-path/remote_path_e4NeK30zS0CFZYvfW2gzjQ#2#1#K85a3uMFS6a2mj1b2AY7Tg
[2026-08-20 03:30:03 UTC]   481B STANDARD opensearch-poc-001/segment/remote-index-path/remote_path_xc_zrs1xQX6mUQbN7aDbmA#2#1#JEZnrg-BRri-EKBwInz00Q
[2026-08-20 03:29:12 UTC]   261B STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/global-metadata/coordination__9223372036854775804__9223370249658223631__1
[2026-08-20 03:29:11 UTC]    32B STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/global-metadata/custom--index-graveyard__9223372036854775806__9223370249658224071__1
[2026-08-20 03:29:11 UTC]   507B STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/global-metadata/custom--repositories__9223372036854775806__9223370249658224073__1
[2026-08-20 03:29:11 UTC]    39B STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/global-metadata/settings__9223372036854775806__9223370249658224089__1
[2026-08-20 03:29:11 UTC]    40B STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/global-metadata/templates__9223372036854775806__9223370249658224075__1
[2026-08-20 03:35:03 UTC] 3.7KiB STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/manifest/manifest__9223372036854775805__9223372036854775759__C__9223370249657872395__4
[2026-08-20 03:35:03 UTC] 3.7KiB STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/manifest/manifest__9223372036854775805__9223372036854775759__P__9223370249657872464__4
[2026-08-20 03:33:43 UTC] 3.7KiB STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/manifest/manifest__9223372036854775805__9223372036854775760__C__9223370249657952725__4
[2026-08-20 03:33:43 UTC] 3.7KiB STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/manifest/manifest__9223372036854775805__9223372036854775760__P__9223370249657952808__4
[2026-08-20 03:33:42 UTC] 3.7KiB STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/manifest/manifest__9223372036854775805__9223372036854775761__C__9223370249657953004__4
[2026-08-20 03:33:42 UTC] 3.7KiB STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/manifest/manifest__9223372036854775805__9223372036854775761__P__9223370249657953054__4
[2026-08-20 03:33:42 UTC] 3.7KiB STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/manifest/manifest__9223372036854775805__9223372036854775762__C__9223370249657953161__4
[2026-08-20 03:33:42 UTC] 3.7KiB STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/manifest/manifest__9223372036854775805__9223372036854775762__P__9223370249657953249__4
[2026-08-20 03:33:42 UTC] 3.7KiB STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/manifest/manifest__9223372036854775805__9223372036854775763__C__9223370249657953474__4
[2026-08-20 03:33:42 UTC] 3.7KiB STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/manifest/manifest__9223372036854775805__9223372036854775763__P__9223370249657953545__4
[2026-08-20 03:33:38 UTC] 3.4KiB STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/manifest/manifest__9223372036854775805__9223372036854775764__C__9223370249657957295__4
[2026-08-20 03:33:38 UTC] 3.4KiB STANDARD opensearch-poc-001/state/b3BlbnNlYXJjaC1wb2MtMDAx/cluster-state/XJUfA7mGTPusZ1qxrAeR9w/manifest/manifest__9223372036854775805__9223372036854775764__P__9223370249657957355__4
[2026-08-20 03:29:12 UTC]   379B STANDARD opensearch-poc-001/translog/remote-index-path/remote_path_6EN_uPXNR-i4u0TGeblgPA#2#1#PKtXtJVaTfOef7wvyhqhyA
[2026-08-20 03:33:42 UTC]   379B STANDARD opensearch-poc-001/translog/remote-index-path/remote_path_FmfgUgKwRuG9_CxWYXbrkw#4#1#qGHvtrCqRaGGm-dR3lkkzw
[2026-08-20 03:29:16 UTC]   379B STANDARD opensearch-poc-001/translog/remote-index-path/remote_path_Hvsn3g4mSDqPgSZQBvRWcw#2#1#3VKqsPifR_axWRn4WTkSQw
[2026-08-20 03:29:12 UTC]   379B STANDARD opensearch-poc-001/translog/remote-index-path/remote_path_R6jVVl-kQ9ShZdwVVP5a_g#2#1#9dnSkO04STSrPKz0xWl8tg
[2026-08-20 03:29:16 UTC]   379B STANDARD opensearch-poc-001/translog/remote-index-path/remote_path_YjQUpXO9Sz2Tv1jzaoE9Lw#2#1#5ImiAf2cS4Gt3-mEOI9OaA
[2026-08-20 03:29:22 UTC]   379B STANDARD opensearch-poc-001/translog/remote-index-path/remote_path_e4NeK30zS0CFZYvfW2gzjQ#2#1#7mrEAIKjQFO2Om-ct6rcmw
[2026-08-20 03:30:03 UTC]   379B STANDARD opensearch-poc-001/translog/remote-index-path/remote_path_xc_zrs1xQX6mUQbN7aDbmA#2#1#fH7np7NnQCq6pVyix6KsZw
[2026-08-20 03:34:27 UTC]    60B STANDARD r10110101110000/opensearch-poc-001/translog/e4NeK30zS0CFZYvfW2gzjQ/0/translog/metadata/metadata__9223372036854775806__9223372036854775803__9223370249657908599__837651239__9223372036854775804__1__1
[2026-08-20 03:29:23 UTC]    56B STANDARD r10110101110000/opensearch-poc-001/translog/e4NeK30zS0CFZYvfW2gzjQ/0/translog/metadata/metadata__9223372036854775806__9223372036854775804__9223370249658212419__837651239__9223372036854775804__1__1
[2026-08-20 03:29:23 UTC]    56B STANDARD r10110101110000/opensearch-poc-001/translog/e4NeK30zS0CFZYvfW2gzjQ/0/translog/metadata/metadata__9223372036854775806__9223372036854775805__9223370249658212722__837651239__9223372036854775805__1__1
[2026-08-20 03:29:22 UTC]    55B STANDARD w00110110111110/opensearch-poc-001/translog/e4NeK30zS0CFZYvfW2gzjQ/0/translog/data/1/translog-2.tlog
[2026-08-20 03:29:23 UTC]   199B STANDARD w00110110111110/opensearch-poc-001/translog/e4NeK30zS0CFZYvfW2gzjQ/0/translog/data/1/translog-3.tlog
[2026-08-20 03:34:27 UTC]    55B STANDARD w00110110111110/opensearch-poc-001/translog/e4NeK30zS0CFZYvfW2gzjQ/0/translog/data/1/translog-4.tlog
[2026-08-20 03:29:17 UTC]    55B STANDARD y00110100011011/opensearch-poc-001/translog/Hvsn3g4mSDqPgSZQBvRWcw/0/translog/data/1/translog-2.tlog
[2026-08-20 03:33:38 UTC]   251B STANDARD y00110100011011/opensearch-poc-001/translog/Hvsn3g4mSDqPgSZQBvRWcw/0/translog/data/1/translog-3.tlog
[2026-08-20 03:38:42 UTC]    55B STANDARD y00110100011011/opensearch-poc-001/translog/Hvsn3g4mSDqPgSZQBvRWcw/0/translog/data/1/translog-4.tlog
[2026-08-20 03:34:17 UTC] 2.9KiB STANDARD y10111101100000/opensearch-poc-001/segment/R6jVVl-kQ9ShZdwVVP5a_g/0/segments/metadata/metadata__9223372036854775806__9223372036854775803__9223372036854775797__9223372036854775796__-1151650581__9223370249657918500__2
[2026-08-20 03:29:15 UTC] 2.9KiB STANDARD y10111101100000/opensearch-poc-001/segment/R6jVVl-kQ9ShZdwVVP5a_g/0/segments/metadata/metadata__9223372036854775806__9223372036854775804__9223372036854775799__9223372036854775797__-1151650581__9223370249658220172__2
[2026-08-20 03:29:15 UTC] 2.6KiB STANDARD y10111101100000/opensearch-poc-001/segment/R6jVVl-kQ9ShZdwVVP5a_g/0/segments/metadata/metadata__9223372036854775806__9223372036854775804__9223372036854775799__9223372036854775798__-1151650581__9223370249658220343__2
[2026-08-20 03:29:15 UTC] 2.3KiB STANDARD y10111101100000/opensearch-poc-001/segment/R6jVVl-kQ9ShZdwVVP5a_g/0/segments/metadata/metadata__9223372036854775806__9223372036854775804__9223372036854775799__9223372036854775799__-1151650581__9223370249658220554__2
[2026-08-20 03:29:15 UTC] 2.1KiB STANDARD y10111101100000/opensearch-poc-001/segment/R6jVVl-kQ9ShZdwVVP5a_g/0/segments/metadata/metadata__9223372036854775806__9223372036854775804__9223372036854775801__9223372036854775800__-1151650581__9223370249658220733__2
[2026-08-20 03:29:14 UTC] 1.8KiB STANDARD y10111101100000/opensearch-poc-001/segment/R6jVVl-kQ9ShZdwVVP5a_g/0/segments/metadata/metadata__9223372036854775806__9223372036854775804__9223372036854775801__9223372036854775801__-1151650581__9223370249658220882__2
[2026-08-20 03:29:14 UTC] 1.5KiB STANDARD y10111101100000/opensearch-poc-001/segment/R6jVVl-kQ9ShZdwVVP5a_g/0/segments/metadata/metadata__9223372036854775806__9223372036854775804__9223372036854775801__9223372036854775802__-1151650581__9223370249658221067__2
[2026-08-20 03:29:14 UTC] 1.2KiB STANDARD y10111101100000/opensearch-poc-001/segment/R6jVVl-kQ9ShZdwVVP5a_g/0/segments/metadata/metadata__9223372036854775806__9223372036854775804__9223372036854775802__9223372036854775803__-1151650581__9223370249658221232__2
[2026-08-20 03:29:14 UTC]  1001B STANDARD y10111101100000/opensearch-poc-001/segment/R6jVVl-kQ9ShZdwVVP5a_g/0/segments/metadata/metadata__9223372036854775806__9223372036854775804__9223372036854775803__9223372036854775804__-1151650581__9223370249658221395__2
[2026-08-20 03:29:14 UTC]   724B STANDARD y10111101100000/opensearch-poc-001/segment/R6jVVl-kQ9ShZdwVVP5a_g/0/segments/metadata/metadata__9223372036854775806__9223372036854775804__9223372036854775803__9223372036854775805__-1151650581__9223370249658221799__2
[2026-08-20 03:29:13 UTC]   724B STANDARD y10111101100000/opensearch-poc-001/segment/R6jVVl-kQ9ShZdwVVP5a_g/0/segments/metadata/metadata__9223372036854775806__9223372036854775804__9223372036854775804__9223372036854775806__-1151650581__9223370249658221925__2

$ ./mc ls m/search-opensearch-remote-store/
[2026-08-20 03:55:38 UTC]     0B 010110111101101/
[2026-08-20 03:55:38 UTC]     0B 1eOzz3odRP8/
[2026-08-20 03:55:38 UTC]     0B 210110011010110/
[2026-08-20 03:55:38 UTC]     0B 410010000010111/
[2026-08-20 03:55:38 UTC]     0B 910100110010110/
[2026-08-20 03:55:38 UTC]     0B B00100110110000/
[2026-08-20 03:55:38 UTC]     0B CLC_HAWyCvg/
[2026-08-20 03:55:38 UTC]     0B F01001101111101/
[2026-08-20 03:55:38 UTC]     0B F4IlRZDg3-E/
[2026-08-20 03:55:38 UTC]     0B FDc6Y8ff6f8/
[2026-08-20 03:55:38 UTC]     0B K10011011101001/
[2026-08-20 03:55:38 UTC]     0B K11101011110111/
[2026-08-20 03:55:38 UTC]     0B M10100111010100/
[2026-08-20 03:55:38 UTC]     0B N00111001101011/
[2026-08-20 03:55:38 UTC]     0B N10010111101011/
[2026-08-20 03:55:38 UTC]     0B N11100011111000/
[2026-08-20 03:55:38 UTC]     0B N11100110110111/
[2026-08-20 03:55:38 UTC]     0B PQNy_8iZAD0/
[2026-08-20 03:55:38 UTC]     0B Q11011011100101/
[2026-08-20 03:55:38 UTC]     0B U00101111001011/
[2026-08-20 03:55:38 UTC]     0B X10100111001001/
[2026-08-20 03:55:38 UTC]     0B X11001111010100/
[2026-08-20 03:55:38 UTC]     0B Z00111101110000/
[2026-08-20 03:55:38 UTC]     0B _10001110000100/
[2026-08-20 03:55:38 UTC]     0B _11110100011001/
[2026-08-20 03:55:38 UTC]     0B _JdJd1ykuiM/
[2026-08-20 03:55:38 UTC]     0B d10110101110111/
[2026-08-20 03:55:38 UTC]     0B g10011100010110/
[2026-08-20 03:55:38 UTC]     0B iyxlDQkxQVU/
[2026-08-20 03:55:38 UTC]     0B l11001100101100/
[2026-08-20 03:55:38 UTC]     0B o01001011010000/
[2026-08-20 03:55:38 UTC]     0B opensearch-poc-001/
[2026-08-20 03:55:38 UTC]     0B r10110101110000/
[2026-08-20 03:55:38 UTC]     0B w00110110111110/
[2026-08-20 03:55:38 UTC]     0B y00110100011011/
[2026-08-20 03:55:38 UTC]     0B y10111101100000/
7

@Camusensei I did some research and testing. The OpenSearch documentation refers to the following options:

cluster.remote_store.index_metadata.path_type: FIXED
cluster.remote_store.index.path.type: fixed

After setting both options, MinIO now shows the expected directory structure:

bash-5.1# mc ls local/search-opensearch-remote-store/                  
[2026-08-21 18:25:58 UTC]     0B opensearch-poc-002/
bash-5.1# mc ls local/search-opensearch-remote-store/opensearch-poc-002
[2026-08-21 18:26:05 UTC]     0B segment/
[2026-08-21 18:26:05 UTC]     0B state/
[2026-08-21 18:26:05 UTC]     0B translog/
8

Sorry for the delayed response, I did not see you replied (I haven’t started catching up to my e-mails yet)

Thank you so much once again, I would probably never have tried cluster.remote_store.index.path.type=FIXED because the documentation says it’s the default but it apparently isn’t!

This closes the topic completely, thank you.