Snapshot policy fails

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OS: 2.10
OD: 2.10

Describe the issue:
The snapshot policy keep on failing with partial status, but when i run snapshot manually it works, what could be the cause.

Configuration:

policy:

{
“description”: “Daily snapshot policy”,
“creation”: {
“schedule”: {
“cron”: {
“expression”: “0 6 * * *”,
“timezone”: “UTC”
}
}
},
“deletion”: {
“schedule”: {
“cron”: {
“expression”: “0 1 * * ",
“timezone”: “America/Los_Angeles”
}
},
“condition”: {
“max_age”: “3d”,
“max_count”: 21,
“min_count”: 7
},
“time_limit”: “1h”
},
“snapshot_config”: {
“date_format”: “yyyy-MM-dd-HH:mm”,
“timezone”: “America/Los_Angeles”,
“indices”: "device-”,
“repository”: “gcs-repo”,
“ignore_unavailable”: “true”,
“include_global_state”: “false”,
“partial”: “true”,
“metadata”: {
“any_key”: “any_value”
}
}
}

Relevant Logs or Screenshots:

error status

[
{
“index”: “device-2023-12-27”,
“index_uuid”: “device-2023-12-27”,
“shard_id”: 3,
“reason”: “AccessControlException[access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")]”,
“node_id”: “9qXQNx1IQey2bQmemS5t6A”,
“status”: “INTERNAL_SERVER_ERROR”
}
]

can someone please help me on this.

2

Hey @arun_udaiyar

Looks like you have a permission issue.

Not sure but maybe CRON doesnt have access. perhaps check the permission on the repo where the sanpshots are located.

3

Thanks @Gsmitt
After adding few more permission at bucket side its works fine.

but the detetion is failing without message, any advice on this

image
image1737×302 23.1 KB

4

Hey @arun_udaiyar

I noticed your CRON timezone is "“timezone”: “UTC”".
But you have snapshots timezone "“timezone”: “America/Los_Angeles”,"

I would assume for deletion either its still might be a permission issue or Date/Time. Not 100% sure thou.

EDIT: Do you see anything in the logs that pertain to this issue of not deleteing your snapshot?

5

@arun_udaiyar Can you explain what you had to change on the bucket side to enable this permission? I am having the same issue where snapshot policies are failing to create/delete and there is no information printed to the log.