Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
3.0.0/3.0.0/opensuse leap 15.4/vivaldi
Describe the issue:
After update to version 3.0.0 from 2.19.1 , i got a lot of slack notification about error creating/deleting snapshot.
there was no changes on s3 side or within opensearch configmaps/secrets
Configuration:
opensearch is running as statefulset on rke2 kubernetes cluster
Relevant Logs or Screenshots:
when i initialize snapshot in dashboard, in pods i can see:
central-workload-1:~ # k logs -n opensearch-stack opensearch-cluster-master-3 | grep "derrr"
Defaulted container "opensearch" out of: opensearch, configfile (init)
[2025-06-09T13:17:53,919][INFO ][o.o.s.SnapshotsService ] [opensearch-cluster-master-3] snapshot [OTC_S3:derrr/kS5-qMh5S3SOEUb8a5nJTg] started
[2025-06-09T13:17:53,959][WARN ][o.o.s.SnapshotShardsService] [opensearch-cluster-master-3] [[jump.wvsc-2025.06.07][0]][OTC_S3:derrr/kS5-qMh5S3SOEUb8a5nJTg] failed to snapshot shard
org.opensearch.core.index.snapshots.IndexShardSnapshotFailedException: Failed to write shard level snapshot metadata for [derrr/kS5-qMh5S3SOEUb8a5nJTg] to [index-xy5-9TAKSmmfJnpl59hXxg]
[2025-06-09T13:17:54,994][WARN ][o.o.s.SnapshotsService ] [opensearch-cluster-master-3] [OTC_S3:derrr/kS5-qMh5S3SOEUb8a5nJTg] failed to finalize snapshot
org.opensearch.snapshots.SnapshotException: [OTC_S3:derrr/kS5-qMh5S3SOEUb8a5nJTg] failed to update snapshot in repository
when i tried remove snapshot via dashboard. i got:
central-workload-1:~ # k logs -n opensearch-stack opensearch-cluster-master-3 | grep "otgxhxm9"
Defaulted container "opensearch" out of: opensearch, configfile (init)
[2025-06-09T13:29:04,867][INFO ][o.o.s.SnapshotsService ] [opensearch-cluster-master-3] deleting snapshots [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9] from repository [OTC_S3]
[2025-06-09T13:29:12,075][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.dmag-2025.05.21][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-jctahPKSSAeoGUMEaB1OKA]
[2025-06-09T13:29:12,079][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.dmag-2025.05.25][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-aF_cj6E0R_KvdUdvwwMDsg]
[2025-06-09T13:29:12,099][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.dmag-2025.05.22][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-EQpPx1SgQBOuQQ87Uf0HoA]
[2025-06-09T13:29:12,109][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.subingen-2025.05.21][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-cCZBJtRsQTil1R12evD3gg]
[2025-06-09T13:29:12,112][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.wvsc-2025.05.25][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-3wq_HQ2eQsKWnVBSBLLwcA]
[2025-06-09T13:29:12,113][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.sissach-2025.05.21][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-M_0oNYmVTMe8_ZWHF7_J4w]
[2025-06-09T13:29:12,116][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.sissach-2025.05.23][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-g46dse1jTc2StD0oNeEKAQ]
[2025-06-09T13:29:12,120][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.langnau-2025.05.23][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-woWZ3el3TPeqPWi7BccK8Q]
[2025-06-09T13:29:12,136][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.dmag-2025.05.20][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-zdMzmjAiTFiz7nFpN_8TDg]
[2025-06-09T13:29:12,145][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.langnau-2025.05.19][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-cc78OG7bRp-qZ8Xwd9Gyiw]
[2025-06-09T13:29:12,143][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.langnau-2025.05.20][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-XHwRcER_TRqH8tL1oJ1fbg]
[2025-06-09T13:29:12,154][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.ettenheim-2025.05.19][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-MWkDF7yJSlWOUnGBDqGShQ]
[2025-06-09T13:29:12,156][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.sissach-2025.05.22][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-UlxM9OvsSBiMtOlXvfR6XA]
[2025-06-09T13:29:12,159][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.vallingby-2025.05.19][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-xvn7UAoLRcGGXkUdtfxfdA]
[2025-06-09T13:29:12,163][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.wvsc-2025.05.22][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-49btUyF1TQq7ZW8x2KAYAg]
[2025-06-09T13:29:12,165][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.vallingby-2025.05.23][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-u1vwmJbGTne898QG1ZMVaQ]
[2025-06-09T13:29:12,165][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [security-auditlog-2024.05.09][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-FoY1cn9OSROCyngssfIP-Q]
[2025-06-09T13:29:12,167][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.vallingby-2025.05.24][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-bqfhYIAUQMKNa8i0Ndrf1Q]
[2025-06-09T13:29:12,169][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.wvsc-2025.05.24][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-HZyM6mIGRteHGy-NR3Qm6Q]
[2025-06-09T13:29:12,171][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.ettenheim-2025.05.23][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-WgypH4I4TVGcC6Xcyqc8HA]
[2025-06-09T13:29:12,173][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.wvsc-2025.05.20][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-Lwh6LZKLTDSaMXCwam3qiw]
[2025-06-09T13:29:12,177][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.vallingby-2025.05.25][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-0r0A3kf3R7OEjXLr3jB_7A]
[2025-06-09T13:29:12,179][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.sissach-2025.05.24][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-EGh82oV6RUmoZFDslsLuNA]
[2025-06-09T13:29:12,177][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.dmag-2025.05.23][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-eqxVDLDaR_Czzf99A5eR7g]
[2025-06-09T13:29:12,179][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.sissach-2025.05.20][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-rtySmuUaQ3CBeJ3Bp5O65g]
[2025-06-09T13:29:12,176][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.subingen-2025.05.25][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-Ff9y11EzROeR9e48SdlY3w]
[2025-06-09T13:29:12,176][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.singen-2025.05.23][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-f65x7GyiTWa6g5-aEAUa8w]
[2025-06-09T13:29:12,174][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.ettenheim-2025.05.22][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-BuChzb4zSq-eAz0cQQSTGw]
[2025-06-09T13:29:12,176][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.ettenheim-2025.05.21][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-ttJeTDOUQZ2j8TgyDiyclQ]
[2025-06-09T13:29:12,177][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.coteau-2025.05.22][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-NB46VT04Tgqh4Jo4LtC21Q]
[2025-06-09T13:29:12,174][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.singen-2025.05.19][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-d6rGCBxHSqSJqxJXuFMORw]
[2025-06-09T13:29:12,183][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.langnau-2025.05.24][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-mw6UpxtpSd2XQcRSRJNNaw]
[2025-06-09T13:29:12,184][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.vallingby-2025.05.20][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-OgE6o0XqRMmsMlK6u-zSiw]
[2025-06-09T13:29:12,183][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.langnau-2025.05.25][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-QBEpcMm6SWeDaHVAJq9Cxg]
[2025-06-09T13:29:12,189][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.ettenheim-2025.05.20][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-lIkrHCiNSB-Hv8axIwoHtA]
[2025-06-09T13:29:12,187][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.wvsc-2025.05.23][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-R2CBEPtDRUWyiJo0XH-L6w]
[2025-06-09T13:29:12,185][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [security-auditlog-2024.05.08][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-phzg73cMQIy9cdkqFDMRrw]
[2025-06-09T13:29:12,187][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.singen-2025.05.24][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-tIDtGFjRQS2jC9vp5_dJcQ]
[2025-06-09T13:29:12,191][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.vallingby-2025.05.21][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-JChuLFtZTVyc0vbMeyAzbA]
[2025-06-09T13:29:12,184][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [security-auditlog-2024.05.12][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-wMejO4rlQoycqhvs_J8img]
[2025-06-09T13:29:12,199][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [security-auditlog-2024.05.14][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-0XAqU_L0TA6clwDMOuQFUw]
[2025-06-09T13:29:12,207][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.coteau-2025.05.24][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-Fpcjyx8AQu2hGieSynOddA]
[2025-06-09T13:29:12,212][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [security-auditlog-2024.05.10][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-VbbpdmU9TTyl30Ld8Lf_mQ]
[2025-06-09T13:29:12,212][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.wvsc-2025.05.19][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-XFua1KOISrCfUgSBJFPeWA]
[2025-06-09T13:29:12,212][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [security-auditlog-2024.05.13][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-mbFENHxqRiinDMkhTmmKKg]
[2025-06-09T13:29:12,223][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.dmag-2025.05.19][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-y7mTHN2VToWSOds6TrsNdA]
[2025-06-09T13:29:12,240][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.subingen-2025.05.23][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-LpQnBOEjQSyUWx0cc39tPg]
[2025-06-09T13:29:12,242][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.singen-2025.05.20][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-6CxDzRn3QeGsLS3l7CUI1Q]
[2025-06-09T13:29:12,271][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.singen-2025.05.21][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-9ikqHrNdQhG9ViRzLP5z7Q]
[2025-06-09T13:29:12,277][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.langnau-2025.05.21][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-m1wLeDoUSR-CZG1jRSRaDw]
[2025-06-09T13:29:12,325][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.ettenheim-2025.05.24][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-f45_CIvfQ_WcTtlLl4Jr6A]
[2025-06-09T13:29:12,333][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.coteau-2025.05.20][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-LGpqT7P_RcqM79C3sIhS3g]
[2025-06-09T13:29:12,343][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.wvsc-2025.05.21][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-PwNbtDxcTXu6-WlaTqJfyA]
[2025-06-09T13:29:12,368][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.langnau-2025.05.22][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-jbPp5N7oRPKVExnRJgmYWA]
[2025-06-09T13:29:12,374][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.subingen-2025.05.22][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-a9VOqCz7TbOXkNNb0078Fw]
[2025-06-09T13:29:12,381][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.coteau-2025.05.21][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-7wnRPa0QTW-5iWxby20dog]
[2025-06-09T13:29:12,406][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.coteau-2025.05.25][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-DM_vNZ4xThKyrjokkXhSdQ]
[2025-06-09T13:29:12,407][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.subingen-2025.05.24][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-7j5mEmqjSZO7zcO8h0a6-w]
[2025-06-09T13:29:13,528][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.coteau-2025.05.23][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-LYXplBzjQi2uLst-u_wi-w]
[2025-06-09T13:29:13,553][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.vallingby-2025.05.22][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-2tBm5tM0QXegytuSlV11iQ]
[2025-06-09T13:29:13,565][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.sissach-2025.05.19][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-tB7BXn1sSUyzB9eoO4xPzQ]
[2025-06-09T13:29:13,714][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.dmag-2025.05.24][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-Y9ETlutWRWmBZUiiqE3GSg]
[2025-06-09T13:29:13,837][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.singen-2025.05.25][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-7sndlLrtTLaO3n3ENdoEVg]
[2025-06-09T13:29:14,097][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.sissach-2025.05.25][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-WFgI8mbsT5-WMPjHEvcWIA]
[2025-06-09T13:29:14,170][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.subingen-2025.05.19][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-On-pbo0URdexhQEnkWY7AQ]
[2025-06-09T13:29:14,200][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [security-auditlog-2024.05.11][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-mZ6yHWJQQV6Z0hjR5AQKBQ]
[2025-06-09T13:29:14,305][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.ettenheim-2025.05.25][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-EU4X050SQ_yn9IY8u-f-MQ]
[2025-06-09T13:29:14,344][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.subingen-2025.05.20][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-C_JN4w8fR5G5FLdmhGMH6A]
[2025-06-09T13:29:14,403][WARN ][o.o.r.b.BlobStoreRepository] [opensearch-cluster-master-3] [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] failed to delete shard data for shard [jump.singen-2025.05.22][0]
org.opensearch.repositories.RepositoryException: [OTC_S3] Failed to finalize snapshot deletion [daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9/DKN_wpHMT1iKEI5PHO4dEA] with shard index [index-iA2fYY2zSGWiRYLZ7DLhhg]
central-workload-1:~ # k logs -n opensearch-stack opensearch-cluster-master-1 | grep "otgxhxm9"
Defaulted container "opensearch" out of: opensearch, configfile (init)
[2025-06-09T13:29:17,438][WARN ][r.suppressed ] [opensearch-cluster-master-1] path: /_snapshot/OTC_S3/daily-jump-snapshots-2025-05-25t18%3A00%3A50-otgxhxm9, params: {repository=OTC_S3, snapshot=daily-jump-snapshots-2025-05-25t18:00:50-otgxhxm9}
connection to s3 including access key i think its working fine ( in april i had to renew keystore )..
thanks for any hint.
BR
L