Right Cleanup Policy

[kolesaev]

Hello,

I still can’t apply a right cleanup policy. My current policy after a re-applying waits 32 days and removes everything older about 2 days.
Here it is:

{
    "policy": {
        "policy_id": "cleanup-fluent-bit",
        "description": "Maintains the indices open by 30 days, then closes those and delete indices after 30 days",
        "default_state": "ReadWrite",
        "states": [
            {
                "name": "ReadWrite",
                "actions": [
                    {
                        "read_write": {}
                    }
                ],
                "transitions": [
                    {
                        "state_name": "ReadOnly",
                        "conditions": {
                            "min_index_age": "30d"
                        }
                    }
                ]
            },
            {
                "name": "ReadOnly",
                "actions": [
                    {
                        "read_only": {}
                    }
                ],
                "transitions": [
                    {
                        "state_name": "Delete",
                        "conditions": {
                            "min_index_age": "32d"
                        }
                    }
                ]
            },
            {
                "name": "Delete",
                "actions": [
                    {
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "fluent-bit"
                ],
                "priority": 100
            }
        ]
    }
}

Before it ReadOnly state was 2 days and i thought that raise to 30 will keep at least 30 days.

My Terraform config of the Cluster is

resource "aws_opensearch_domain" "cluster" {
  domain_name    = substr("${var.k8s_cluster_name}-opensearch", 0, 28)
  engine_version = "${var.cluster_config.engine}"

  cluster_config {
    instance_type          = var.cluster_config.instance_type
    instance_count         = var.cluster_config.instance_count
    zone_awareness_enabled = var.cluster_config.zone_awareness_enabled
  }

  ebs_options {
    ebs_enabled = var.cluster_config.ebs_options.ebs_enabled
    volume_type = var.cluster_config.ebs_options.volume_type
    volume_size = var.cluster_config.ebs_options.volume_size
    iops = var.cluster_config.ebs_options.iops
    throughput = var.cluster_config.ebs_options.throughput
  }
  
  vpc_options {
    subnet_ids = [var.private_subnets[0]]
    security_group_ids = [aws_security_group.opensearch.id]
  }

  advanced_options = local.advanced_options

  access_policies = <<CONFIG
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "es:*",
            "Principal": "*",
            "Effect": "Allow",
            "Resource": "arn:aws:es:${var.cluster_region}:${var.account_id}:domain/${var.k8s_cluster_name}-opensearch/*"
        }
    ]
}
CONFIG

  log_publishing_options {
    enabled   = false
    log_type  = "INDEX_SLOW_LOGS"
    cloudwatch_log_group_arn = ""
  }

  depends_on = [aws_iam_service_linked_role.opensearch]
}

Engine is Elasticsearch_7.10

I also would like to apply any plugin which would give us possibility to make exports of all records from an index to a json format and add it as a step to my cleanup policy as well.

Please anybody give me an advice or a working policy configuration.

[kolesaev]

i am trying to move to OpenSearch_2.5 engine

upd. Still the same

[kolesaev]

We use AWS OpenSearch 2.5.

Current policy is

{
    "policy": {
        "policy_id": "cleanup-fluent-bit",
        "description": "Policy that deletes indicies older than 30 days.",
        "default_state": "open",
        "states": [
            {
                "name": "open",
                "actions": [
                    {
                        "open": {}
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "30d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "fluent-bit"
                ],
                "priority": 100
            }
        ]
    }
}

But it keeps only about two days of indicies

[Gsmitt]

Hey @kolesaev

Not sure what up with that policy but here is an example of my lab. It holds 3 days, and deletes.

{
    "policy": {
        "policy_id": "metricbeat",
        "description": "delete  index after 3 days",
        "last_updated_time": 1685153715145,
        "schema_version": 17,
        "error_notification": null,
        "default_state": "current",
        "states": [
            {
                "name": "current",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "open": {}
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "3d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "metricbeat-*"                   
                ],
                "priority": 200,
                "last_updated_time": 1682391676151
            }
        ]
    }
}