Cannot acquire lock for snapshot management job

sab · April 1, 2026, 9:31am

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 3.4.0

Describe the issue:

Hello,

Since our upgrade from Opensearch 2.17 to 3.4, scheduling of snapshot dosen’t work.
I can’t find any workaround to fix this problem … Any idea, or help about this issue ?

Configuration:

[DC01] admin@myserver ~ $ curl -s -XGET -u : --negotiate “https://$(hostname --fqdn):9200/_snapshot/s3-news-backup” | jq
{
“s3-news-backup”: {
“type”: “s3”,
“settings”: {
“bucket”: “prod-opensearch02-bsa-backup”,
“base_path”: “s3-news-backup”
}
}
}
[DC01] admin@myserver ~ $ curl -s -XGET -u : --negotiate “https://$(hostname --fqdn):9200/_plugins/_sm/policies/s3-backup” | jq
{
“_id”: “s3-backup-sm-policy”,
“_version”: 1,
“_seq_no”: 840427,
“_primary_term”: 12,
“sm_policy”: {
“name”: “s3-backup”,
“description”: “News daily snapshot policy”,
“schema_version”: 24,
“creation”: {
“schedule”: {
“cron”: {
“expression”: “30 3 * * *”,
“timezone”: “Europe/Paris”
}
},
“time_limit”: “4h”
},
“deletion”: {
“schedule”: {
“cron”: {
“expression”: “42 0 * * ",
“timezone”: “Europe/Paris”
}
},
“condition”: {
“max_age”: “31d”,
“min_count”: 30,
“max_count”: 42
},
“time_limit”: “4h”
},
“snapshot_config”: {
“indices”: "news”,
“ignore_unavailable”: “true”,
“timezone”: “Europe/Paris”,
“include_global_state”: “false”,
“date_format”: “yyyy-MM-dd-HH:mm”,
“repository”: “s3-backup”,
“date_format_timezone”: “Europe/Paris”,
“partial”: “false”
},
“schedule”: {
“interval”: {
“start_time”: 1774521131053,
“period”: 1,
“unit”: “Minutes”
}
},
“enabled”: true,
“last_updated_time”: 1774521131053,
“enabled_time”: 1774521131053
}
}

Relevant Logs or Screenshots:

[2026-04-01T11:21:11,047][WARN ][o.o.i.s.SMRunner ] [myserver] Cannot acquire lock for snapshot management job s3-backup
[2026-04-01T11:22:11,043][INFO ][o.o.j.t.PluginClient ] [myserver] Running transport action with subject: plugin:org.opensearch.jobscheduler.JobSchedulerPlugin
[2026-04-01T11:22:11,043][INFO ][o.o.s.p.SystemIndexAccessEvaluator] [myserver] No indices:data/read/get permission for user roles to System Indices .opendistro-job-scheduler-lock
[2026-04-01T11:22:11,044][ERROR][o.o.j.u.LockServiceImpl ] [myserver] Exception occurred finding lock
org.opensearch.OpenSearchSecurityException: no permissions for and User [name=plugin:org.opensearch.jobscheduler.JobSchedulerPlugin, backend_roles=, requestedTenant=null]
at org.opensearch.security.filter.SecurityFilter.lambda$apply0$3(SecurityFilter.java:414) [opensearch-security-3.4.0.0.jar:3.4.0.0]
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:508) [opensearch-security-3.4.0.0.jar:3.4.0.0]
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:187) [opensearch-security-3.4.0.0.jar:3.4.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:81) [opensearch-performance-analyzer-3.4.0.0.jar:3.4.0.0]
at org.opensearch.action.support.ActionFilter.apply(ActionFilter.java:67) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:109) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.transport.client.node.NodeClient.executeLocally(NodeClient.java:113) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.transport.client.node.NodeClient.doExecute(NodeClient.java:100) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.transport.client.FilterClient.doExecute(FilterClient.java:83) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.jobscheduler.transport.PluginClient.lambda$doExecute$0(PluginClient.java:56) [opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
at org.opensearch.security.identity.SecurePluginSubject.runAs(SecurePluginSubject.java:52) [opensearch-security-3.4.0.0.jar:3.4.0.0]
at org.opensearch.jobscheduler.transport.PluginClient.doExecute(PluginClient.java:54) [opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.transport.client.support.AbstractClient.get(AbstractClient.java:597) [opensearch-3.4.0.jar:3.4.0]
at org.opensearch.jobscheduler.utils.LockServiceImpl.findLock(LockServiceImpl.java:282) [opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
at org.opensearch.jobscheduler.utils.LockServiceImpl.lambda$acquireLockWithId$0(LockServiceImpl.java:180) [opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.4.0.jar:3.4.0]
at org.opensearch.jobscheduler.utils.LockServiceImpl.createLockIndex(LockServiceImpl.java:102) [opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
at org.opensearch.jobscheduler.utils.LockServiceImpl.acquireLockWithId(LockServiceImpl.java:177) [opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
at org.opensearch.jobscheduler.utils.LockServiceImpl.acquireLock(LockServiceImpl.java:138) [opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invokeSuspend$lambda$0(JobSchedulerUtils.kt:28) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:230) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invokeSuspend(JobSchedulerUtils.kt:28) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invoke(JobSchedulerUtils.kt) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invoke(JobSchedulerUtils.kt) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.retry(OpenSearchExtensions.kt:175) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.retry$default(OpenSearchExtensions.kt:166) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt.acquireLockForScheduledJob(JobSchedulerUtils.kt:27) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.snapshotmanagement.SMRunner$runJob$1.invokeSuspend(SMRunner.kt:80) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-2.2.0.jar:2.2.0-release-294]
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
[2026-04-01T11:22:11,044][ERROR][o.o.i.u.JobSchedulerUtils] [myserver] Failed to acquireLock for job s3-backup-sm-policy
org.opensearch.OpenSearchSecurityException: no permissions for and User [name=plugin:org.opensearch.jobscheduler.JobSchedulerPlugin, backend_roles=, requestedTenant=null]
at org.opensearch.security.filter.SecurityFilter.lambda$apply0$3(SecurityFilter.java:414) ~[?:?]
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:508) ~[?:?]
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:187) ~[?:?]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:81) ~[?:?]
at org.opensearch.action.support.ActionFilter.apply(ActionFilter.java:67) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:109) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.transport.client.node.NodeClient.executeLocally(NodeClient.java:113) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.transport.client.node.NodeClient.doExecute(NodeClient.java:100) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.transport.client.FilterClient.doExecute(FilterClient.java:83) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.jobscheduler.transport.PluginClient.lambda$doExecute$0(PluginClient.java:56) ~[?:?]
at org.opensearch.security.identity.SecurePluginSubject.runAs(SecurePluginSubject.java:52) ~[?:?]
at org.opensearch.jobscheduler.transport.PluginClient.doExecute(PluginClient.java:54) ~[?:?]
at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.transport.client.support.AbstractClient.get(AbstractClient.java:597) ~[opensearch-3.4.0.jar:3.4.0]
at org.opensearch.jobscheduler.utils.LockServiceImpl.findLock(LockServiceImpl.java:282) ~[opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
at org.opensearch.jobscheduler.utils.LockServiceImpl.lambda$acquireLockWithId$0(LockServiceImpl.java:180) ~[opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-3.4.0.jar:3.4.0]
at org.opensearch.jobscheduler.utils.LockServiceImpl.createLockIndex(LockServiceImpl.java:102) ~[opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
at org.opensearch.jobscheduler.utils.LockServiceImpl.acquireLockWithId(LockServiceImpl.java:177) ~[opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
at org.opensearch.jobscheduler.utils.LockServiceImpl.acquireLock(LockServiceImpl.java:138) ~[opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invokeSuspend$lambda$0(JobSchedulerUtils.kt:28) ~[opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:230) ~[opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invokeSuspend(JobSchedulerUtils.kt:28) ~[opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invoke(JobSchedulerUtils.kt) ~[opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invoke(JobSchedulerUtils.kt) ~[opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.retry(OpenSearchExtensions.kt:175) ~[opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.retry$default(OpenSearchExtensions.kt:166) ~[opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt.acquireLockForScheduledJob(JobSchedulerUtils.kt:27) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at org.opensearch.indexmanagement.snapshotmanagement.SMRunner$runJob$1.invokeSuspend(SMRunner.kt:80) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-2.2.0.jar:2.2.0-release-294]
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
[2026-04-01T11:22:11,045][WARN ][o.o.i.s.SMRunner ] [myserver] Cannot acquire lock for snapshot management job s3-backup

pablo · April 1, 2026, 9:59am

@sab Could you share opensearch.yml file?

sab · April 1, 2026, 12:16pm

sure @pablo

######## Configuration ########
cluster.name: prod.search

node.name: opensearchadmin02-p01.dc01
node.roles: [ ingest ]

node.attr.datacenter: dc01
cluster.routing.allocation.awareness.attributes: datacenter
cluster.routing.allocation.awareness.force.zone.values: dc01, dc02

network.host: [site]

cluster.initial_cluster_manager_nodes: [“opensearchmaster02-p01.dc01”, “opensearchmaster02-p02.dc01”, “opensearchmaster02-p03.dc02”, “opensearchmaster02-p04.dc02”, “opensearchmaster02-p05.dc06”]

discovery.seed_hosts: [“opensearchadmin02-p01.dc01”, “opensearchadmin02-p02.dc02”, “opensearchdata02-p01.dc01”, “opensearchdata02-p02.dc02”, “opensearchdata02-p03.dc01”, “opensearchdata02-p04.dc02”, “opensearchmaster02-p01.dc01”, “opensearchmaster02-p02.dc02”, “opensearchmaster02-p03.dc01”, “opensearchmaster02-p04.dc02”, “opensearchmaster02-p05.dc06”, “opensearchfront02-p01.dc01”, “opensearchfront02-p03.dc01”, “opensearchfront02-p02.dc02”, “opensearchfront02-p04.dc02”]

bootstrap.memory_lock: true

plugins.security.kerberos.krb5_filepath: /usr/share/opensearch/config/krb5.conf
plugins.security.kerberos.acceptor_keytab_filepath: merged.keytab
plugins.security.kerberos.acceptor_principal: ‘HTTP/opensearchadmin02-p01.dc01’

plugins.security.ssl.transport.pemcert_filepath: /usr/share/opensearch/config/opensearch-security/certs/opensearchadmin02-p01.dc01.pem
plugins.security.ssl.transport.pemkey_filepath: /usr/share/opensearch/config/opensearch-security/certs/opensearchadmin02-p01.dc01.key
plugins.security.ssl.transport.pemtrustedcas_filepath: /usr/local/share/ca-certificates/pki_infrastructure_ca_sha256.crt

plugins.security.ssl.transport.truststore_filepath: /usr/share/opensearch/jdk/lib/security/cacerts

transport.ssl.enforce_hostname_verification: false
transport.ssl.resolve_hostname: false
plugins.security.ssl.transport.enabled_ciphers:

TLS_AES_128_GCM_SHA256

TLS_AES_256_GCM_SHA384

TLS_CHACHA20_POLY1305_SHA256
plugins.security.ssl.transport.enabled_protocols:

TLSv1.2

TLSv1.3

plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /usr/share/opensearch/config/opensearch-security/certs/opensearchadmin02-p01.dc01.pem
plugins.security.ssl.http.pemkey_filepath: /usr/share/opensearch/config/opensearch-security/certs/opensearchadmin02-p01.dc01.key
plugins.security.ssl.http.pemtrustedcas_filepath: /usr/local/share/ca-certificates/pki_infrastructure_ca_sha256.crt

plugins.security.ssl.http.truststore_filepath: /usr/share/opensearch/jdk/lib/security/cacerts

plugins.security.ssl.http.clientauth_mode: OPTIONAL
plugins.security.ssl.http.enabled_ciphers:

TLS_AES_128_GCM_SHA256

TLS_AES_256_GCM_SHA384

TLS_CHACHA20_POLY1305_SHA256
plugins.security.ssl.http.enabled_protocols:

TLSv1.2

TLSv1.3

plugins.security.nodes_dn: [ obfuscate ]

plugins.security.authcz.admin_dn: [ obfuscate ]

plugins.security.audit.type: log4j
plugins.security.audit.config.log4j.logger_name: osaudit
plugins.security.audit.config.log4j.level: INFO

plugins.security.audit.config.enable_rest: true
plugins.security.audit.config.enable_transport: true
plugins.security.audit.config.disabled_rest_categories: NONE
plugins.security.audit.config.disabled_transport_categories: NONE

plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”]

plugins.security.system_indices.permission.enabled: true
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [“.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opensearch-notifications-”, “.opensearch-notebooks”, “.opensearch-observability”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”]

s3.client.default.endpoint: mys3server.net
s3.client.default.max_retries: 3
s3.client.default.path_style_access: true
s3.client.default.protocol: https
s3.client.default.read_timeout: 50s
s3.client.default.use_throttle_retries: true
s3.client.default.region: us-east-2
prometheus.metric_name.prefix: opensearch_
prometheus.indices: true
prometheus.cluster.settings: true
prometheus.nodes.filter: _local

pablo · April 1, 2026, 1:09pm

@sab Could you add this index .opendistro-job-scheduler-lock to plugins.security.system_indices.indices: and test the snapshot.

sab · April 1, 2026, 3:15pm

@pablo after add add this index .opendistro-job-scheduler-lock to plugins.security.system_indices.indices and restart all the cluster, the error still the same

[2026-04-01T17:06:11,044][INFO ][o.o.j.t.PluginClient     ] [opensearchdata02-p01.dc01] Running transport action with subject: plugin:org.opensearch.jobscheduler.JobSchedulerPlugin
[2026-04-01T17:06:11,045][INFO ][o.o.s.p.SystemIndexAccessEvaluator] [opensearchdata02-p01.dc01] No indices:data/read/get permission for user roles [] to System Indices .opendistro-job-scheduler-lock
[2026-04-01T17:06:11,046][ERROR][o.o.j.u.LockServiceImpl  ] [opensearchdata02-p01.dc01] Exception occurred finding lock
org.opensearch.OpenSearchSecurityException: no permissions for [] and User [name=plugin:org.opensearch.jobscheduler.JobSchedulerPlugin, backend_roles=[], requestedTenant=null]
    at org.opensearch.security.filter.SecurityFilter.lambda$apply0$3(SecurityFilter.java:414) [opensearch-security-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:508) [opensearch-security-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:187) [opensearch-security-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) [opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:109) [opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.transport.client.node.NodeClient.executeLocally(NodeClient.java:113) [opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.transport.client.node.NodeClient.doExecute(NodeClient.java:100) [opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501) [opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.transport.client.FilterClient.doExecute(FilterClient.java:83) [opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.jobscheduler.transport.PluginClient.lambda$doExecute$0(PluginClient.java:56) [opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.security.identity.SecurePluginSubject.runAs(SecurePluginSubject.java:52) [opensearch-security-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.jobscheduler.transport.PluginClient.doExecute(PluginClient.java:54) [opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501) [opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.transport.client.support.AbstractClient.get(AbstractClient.java:597) [opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.jobscheduler.utils.LockServiceImpl.findLock(LockServiceImpl.java:282) [opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.jobscheduler.utils.LockServiceImpl.lambda$acquireLockWithId$0(LockServiceImpl.java:180) [opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-3.4.0.jar:3.4.0]
    at org.opensearch.jobscheduler.utils.LockServiceImpl.createLockIndex(LockServiceImpl.java:102) [opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.jobscheduler.utils.LockServiceImpl.acquireLockWithId(LockServiceImpl.java:177) [opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.jobscheduler.utils.LockServiceImpl.acquireLock(LockServiceImpl.java:138) [opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invokeSuspend$lambda$0(JobSchedulerUtils.kt:28) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:230) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invokeSuspend(JobSchedulerUtils.kt:28) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invoke(JobSchedulerUtils.kt) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invoke(JobSchedulerUtils.kt) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.retry(OpenSearchExtensions.kt:175) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.retry$default(OpenSearchExtensions.kt:166) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt.acquireLockForScheduledJob(JobSchedulerUtils.kt:27) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.snapshotmanagement.SMRunner$runJob$1.invokeSuspend(SMRunner.kt:80) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-2.2.0.jar:2.2.0-release-294]
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
    at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
[2026-04-01T17:06:11,046][ERROR][o.o.i.u.JobSchedulerUtils] [opensearchdata02-p01.dc01] Failed to acquireLock for job s3-backup-sm-policy
org.opensearch.OpenSearchSecurityException: no permissions for [] and User [name=plugin:org.opensearch.jobscheduler.JobSchedulerPlugin, backend_roles=[], requestedTenant=null]
    at org.opensearch.security.filter.SecurityFilter.lambda$apply0$3(SecurityFilter.java:414) ~[?:?]
    at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:508) ~[?:?]
    at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:187) ~[?:?]
    at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) ~[opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.action.support.TransportAction.execute(TransportAction.java:109) ~[opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.transport.client.node.NodeClient.executeLocally(NodeClient.java:113) ~[opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.transport.client.node.NodeClient.doExecute(NodeClient.java:100) ~[opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501) ~[opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.transport.client.FilterClient.doExecute(FilterClient.java:83) ~[opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.jobscheduler.transport.PluginClient.lambda$doExecute$0(PluginClient.java:56) ~[?:?]
    at org.opensearch.security.identity.SecurePluginSubject.runAs(SecurePluginSubject.java:52) ~[?:?]
    at org.opensearch.jobscheduler.transport.PluginClient.doExecute(PluginClient.java:54) ~[?:?]
    at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501) ~[opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.transport.client.support.AbstractClient.get(AbstractClient.java:597) ~[opensearch-3.4.0.jar:3.4.0]
    at org.opensearch.jobscheduler.utils.LockServiceImpl.findLock(LockServiceImpl.java:282) ~[opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.jobscheduler.utils.LockServiceImpl.lambda$acquireLockWithId$0(LockServiceImpl.java:180) ~[opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-3.4.0.jar:3.4.0]
    at org.opensearch.jobscheduler.utils.LockServiceImpl.createLockIndex(LockServiceImpl.java:102) ~[opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.jobscheduler.utils.LockServiceImpl.acquireLockWithId(LockServiceImpl.java:177) ~[opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.jobscheduler.utils.LockServiceImpl.acquireLock(LockServiceImpl.java:138) ~[opensearch-job-scheduler-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invokeSuspend$lambda$0(JobSchedulerUtils.kt:28) ~[opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:230) ~[opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invokeSuspend(JobSchedulerUtils.kt:28) ~[opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invoke(JobSchedulerUtils.kt) ~[opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt$acquireLockForScheduledJob$2.invoke(JobSchedulerUtils.kt) ~[opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.retry(OpenSearchExtensions.kt:175) ~[opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.retry$default(OpenSearchExtensions.kt:166) ~[opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.util.JobSchedulerUtilsKt.acquireLockForScheduledJob(JobSchedulerUtils.kt:27) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at org.opensearch.indexmanagement.snapshotmanagement.SMRunner$runJob$1.invokeSuspend(SMRunner.kt:80) [opensearch-index-management-3.4.0.0.jar:3.4.0.0]
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-2.2.0.jar:2.2.0-release-294]
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
    at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684) [kotlinx-coroutines-core-jvm-1.7.3.jar:?]
[2026-04-01T17:06:11,048][WARN ][o.o.i.s.SMRunner         ] [opensearchdata02-p01.dc01] Cannot acquire lock for snapshot management job s3-backup

pablo · April 1, 2026, 6:31pm

@sab Thanks for testing. The issue is related to the system indices permission.

This setting allows assigning permissions to a regular user to access system indices.

plugins.security.system_indices.permission.enabled: true

In 3.4, there was a permission check implemented, but it only worked for users. The issue is related to the plugin that has no roles assigned.

That’s why you’ll see this line in your logs.

[2026-04-01T11:22:11,043][INFO ][o.o.j.t.PluginClient ] [myserver] Running transport action with subject: plugin:org.opensearch.jobscheduler.JobSchedulerPlugin
[2026-04-01T11:22:11,043][INFO ][o.o.s.p.SystemIndexAccessEvaluator] [myserver] No indices:data/read/get permission for user roles  to System Indices .opendistro-job-scheduler-lock

You have two options to workaround/resolve this issue.

Set plugins.security.system_indices.permission.enabled: to false
Upgrade OpenSearch to version 3.5, where this issue has been fixed, and keep using plugins.security.system_indices.permission.enabled: true.

github.com/opensearch-project/security

Skip hasExplicitIndexPrivilege check for plugin users accessing their own system indices (#5858)

main ← llilyy:lily/plugin-system-index

opened 07:35AM - 15 Dec 25 UTC

llilyy

+411 -190

### Description * Category * Bug fix * Why these changes are required…? * When `isSystemIndexPermissionEnabled` is set to `true`, plugin users are unable to access their own registered system indices. This happens because the `hasExplicitIndexPrivilege` check is performed before the plugin user's own system index access logic, causing the request to be rejected even though the plugin should have access to its own system indices. * This breaks the expected behavior where plugins can manage their own system indices without requiring explicit `system_index` permission. * What is the old behavior before changes and new behavior after changes? * Old behavior: * When `isSystemIndexPermissionEnabled = true`, all system index access (including plugin's own system indices) goes through `hasExplicitIndexPrivilege` check * Plugin users are rejected with "no permissions" error when trying to access their own registered system indices * The plugin user special handling logic at the bottom of `evaluateSystemIndicesAccess()` is never reached because the request is already rejected * New behavior: * Plugin users can access their own registered system indices for non-cluster actions without requiring `hasExplicitIndexPrivilege` check * The `isPluginIndexAction` condition is added to skip the permission check when: * The user is a plugin user * The action is not a cluster-level action * The request only targets the plugin's own registered system indices * Cluster-level actions are still properly restricted for plugin users * A new helper method `getMatchingPluginIndices()` is added to calculate plugin's own system indices once and reuse it, avoiding code duplication ### Issues Resolved https://github.com/opensearch-project/job-scheduler/issues/865 ### Testing **New Integration Tests Added:** - Created `SystemIndexPermissionEnabledTests.java` with `isSystemIndexPermissionEnabled = true` configuration - Tests verify plugin users can access their own system indices when system index permission is enabled: - `testPluginShouldBeAbleToIndexDocumentIntoItsSystemIndex` - `testPluginShouldBeAbleToBulkIndexDocumentIntoItsSystemIndex` - `testPluginShouldBeAbleSearchOnItsSystemIndex` - `testPluginShouldBeAbleGetOnItsSystemIndex` - `testPluginShouldBeAbleUpdateOnItsSystemIndex` - Tests verify plugin users are still properly restricted: - `testPluginShouldNotBeAbleToIndexDocumentIntoSystemIndexRegisteredByOtherPlugin` - `testPluginShouldNotBeAbleToRunClusterActions` - `testPluginShouldNotBeAbleToBulkIndexDocumentIntoMixOfSystemIndexWhereAtLeastOneDoesNotBelongToPlugin` **Existing Tests Verified:** - `SystemIndexTests` - All tests pass (no regression with `isSystemIndexPermissionEnabled = false`) - `SystemIndexAccessEvaluatorTest` - All unit tests pass (no impact on regular user behavior) **Test Execution:** ```bash ./gradlew :integrationTest --tests "org.opensearch.security.systemindex.SystemIndexTests" --tests "org.opensearch.security.systemindex.SystemIndexPermissionEnabledTests" # BUILD SUCCESSFUL ./gradlew :test --tests "org.opensearch.security.privileges.SystemIndexAccessEvaluatorTest" # BUILD SUCCESSFUL ``` ### Check List - [ ] New functionality includes testing - [ ] New functionality has been documented - [ ] New Roles/Permissions have a corresponding security dashboards plugin PR - [ ] API changes companion pull request [created](https://github.com/opensearch-project/opensearch-api-specification/blob/main/DEVELOPER_GUIDE.md) - [ ] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/security/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

I’ve tested both options. No errors in the logs, and the snapshot policy worked.

sab · April 2, 2026, 7:26am

Hello @pablo
plugins.security.system_indices.permission.enabled: to true is a workaround to fix another issue described here :

github.com/opensearch-project/security

[BUG] Access to .opensearch_dashboards Index Blocked

opened 03:26AM - 25 May 25 UTC

inntran

bug triaged

**What is the bug?** The OpenSearch Dashboards service fails to start due to ins…ufficient permissions for the `kibanaserver` user to access system indices such as `.opensearch_dashboards_*`. Despite having the correct role configurations, the user is unable to perform required actions like `indices:admin/aliases`, which is necessary for the service to initialize. When ``` opensearchDashboards.index: ".opensearch_dashboards" ``` is commented out, the OpenSearch Dashboards would use ".kibana" as the index, the OpenSearch Dashboards service can start without issues. **How can one reproduce the bug?** Steps to reproduce the behavior: 1. Configure the `kibanaserver` role with appropriate index permissions (e.g., for `.opensearch_dashboards_*`). 2. Attempt to start OpenSearch Dashboards. 3. Observe that the service fails to start, and review the OpenSearch and Dashboards logs for errors related to insufficient permissions for system indices. **What is the expected behavior?** The OpenSearch Dashboards service should start successfully with the `kibanaserver` user able to perform the necessary actions on system indices such as `.opensearch_dashboards_*`, including `indices:admin/aliases`. **What is your host/environment?** - OS: AlmaLinux 9 - OpenSearch Version: 3.0.0 - OpenSearch Dashboards Version: 3.0.0 - Plugins: OpenSearch Security plugin **Do you have any screenshots?** N/A **Do you have any additional context?** The logs from OpenSearch Dashboards and OpenSearch indicate that the `kibanaserver` user is being blocked from performing the `indices:admin/aliases` action on `.opensearch_dashboards_*` indices. This issue prevents the OpenSearch Dashboards service from starting up. OpenSearch Log Example: ``` [2025-05-25T03:20:28,845][WARN ][o.o.s.p.SystemIndexAccessEvaluator] [osv3test] indices:admin/aliases for '.opensearch_dashboards_1' index is not allowed for a regular user ``` OpenSearch Dashboards Log Example: ``` [error][opensearch][data] security_exception: no permissions for [] and User [name=kibanaserver, backend_roles=[], requestedTenant=null] ``` It appears that the current role configuration does not grant the necessary permissions to access or modify system indices. Adjusting the role permissions or adding specific access rights for system indices may resolve the issue. This issue looks like related to #4439 .

If I set it to false, opensearch-dashboard which will no longer work for most of my users.
Can you tell me if the upgrade to 3.5 also resolves the issue with the .opensearch_dashboards indexes?

This might not be the right place, but I’m also having issues with the repository-s3 plugin since the rolling-restart of my cluster:

{“error”:{“root_cause”:[{“type”:“sdk_client_exception”,“reason”:“sdk_client_exception: IMDS credentials have been disabled by environment variable or system property.”}],“type”:“repository_exception”,“reason”:“[s3-news-backup] Unexpected exception when loading repository data”,“caused_by”:{“type”:“sdk_client_exception”,“reason”:“sdk_client_exception: IMDS credentials have been disabled by environment variable or system property.”}},“status”:500}

Manual snapshot dosen’t work any more with my private s3
Credentials is set via keystore, env var AWS_EC2_METADATA_DISABLED is set to true

[opensearch@opensearchadmin02-p01 ~]$ env |grep AWS
AWS_EC2_METADATA_DISABLED=true
[opensearch@opensearchadmin02-p01 ~]$ opensearch-keystore list
keystore.seed
s3.client.default.access_key
s3.client.default.secret_key

pablo · April 3, 2026, 10:49am

@sab I was only testing your initial issue. The error appears because plugins.security.system_indices.permission.enabled works only with regular users. The plugin clients have no roles assigned, and because of that, they couldn’t access their own system indices.

This has been fixed in 3.5, and permission check is no longer enforced against plugin clients.
Also, this means that plugins.security.system_indices.permission.enabled: can be set to true in 3.5, and it won’t cause issues as it did in 3.4.

Regarding the snapshot, please open a new thread.

sab · April 3, 2026, 11:56am

@pablo no problems, I come back to you in few days after upgrade to confirm if the intials problem has been resolved

pablo · April 3, 2026, 1:05pm

@sab You could consider a small test cluster with OS 3.5 and see if your users can access .opensearch_dashboards index and no permission errors are triggered.

sab · April 3, 2026, 1:08pm

@pablo yes it’s in the pipe, we have dev and staging env

Topic		Replies	Views
Problem with index permissions Security index-management	10	1026	July 26, 2024
What storage service can I use for snapshots? OpenSearch configure	11	1732	August 22, 2022
Restore shared fs snapshots permissions issues 2.x OpenSearch troubleshoot	4	528	August 15, 2022
Security permission issue while restoring snapshot from elastic cluster to opensearch cluster OpenSearch discuss , troubleshoot	13	3692	June 22, 2022
Not able to see snapshot management option in opensearch menu in cluster version 2.3 OpenSearch Dashboards	2	304	June 22, 2023

Cannot acquire lock for snapshot management job

Related topics