Hi,
The logging system consists of fluent-bit → opensearch → opensearch-dashboards deployed in k8s.
Need to clean up disk space in PV of k8s from logs that are older than 3 days.
Versions:
fluent-bit: 3.0.7
opensearch: 2.15.0
opensearch-dasboards: 2.15.0
Describe the issue:
Сreated policy in Index Management does not remove logs.
Configuration:
What could be the problem?
Help, please!
Where could the error be?
Seems the policy is checking if the indexes meet the transition condition which is the creation time is more than 3 days, could you show the complete ISM policy in json format, you can click the View JSON button in the policy detail page:
Certainly:
{
"id": "clean_up",
"seqNo": 76,
"primaryTerm": 1,
"policy": {
"policy_id": "clean_up",
"description": "A sample description of the policy",
"last_updated_time": 1721138293369,
"schema_version": 21,
"error_notification": null,
"default_state": "retention",
"states": [
{
"name": "retention",
"actions": [],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "3d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"delete": {}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"kube*"
],
"priority": 100,
"last_updated_time": 1721138293369
}
]
}
}
This policy works in my local env, could you check the running status by calling explain API:
GET _plugins/_ism/explain/{index_name}
, and even retry the failed state if exists:
POST _plugins/_ism/retry/index_1
{
"state": "retention"
}
. Here are the documentation of ISM APIs: https://opensearch.org/docs/latest/im-plugin/ism/api/#retry-failed-index
{
"kube-2024.07.23": {
"index.plugins.index_state_management.policy_id": "clean_up",
"index.opendistro.index_state_management.policy_id": "clean_up",
"index": "kube-2024.07.23",
"index_uuid": "8hgYHgY7RXqSq3qBPPgC2w",
"policy_id": "clean_up",
"policy_seq_no": 76,
"policy_primary_term": 1,
"index_creation_date": 1721692801006,
"state": {
"name": "retention",
"start_time": 1721693229965
},
"action": {
"name": "transition",
"start_time": 1721693576847,
"index": -1,
"failed": false,
"consumed_retries": 0,
"last_retry_time": 0
},
"step": {
"name": "attempt_transition_step",
"start_time": 1721693576847,
"step_status": "condition_not_met"
},
"retry_info": {
"failed": false,
"consumed_retries": 0
},
"info": {
"message": "Evaluating transition conditions [index=kube-2024.07.23]"
},
"enabled": true
},
"total_managed_indices": 1
}
This response is normal because the index kube-2024.07.23 doesn’t meet the transition condition which is older than 3 days.
ок
how can I delete logs that are older than three days?
Hello John,
Got many issues with those until I managed to understand everything, I understand your pain hehe!
Can you try again the GET _plugins/_ism/explain/{index_name} but with an index that’s not working? Maybe the kube-2024-07-19 instead of the 23 ? (Because it’s normal that the 23 was not erased on the 24th).
Also did you recently change that policy? You can manually do “Remove policy” on every index in Policy managed indexes and reapply again, to be sure new settings are taken into account. After that, you need to wait a good amount of time until it reapply itself and every Job status is back at Running/Evaluating.
Thanks!