I want to archive my old logs to s3 bucket. Currently I'm using docker-compose for the opensearch setup. Can anyone help on this please

OpenSearch
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
v 2.8.0

Describe the issue:
No issue. Looking for option to archive old logs to s3 bucket

Configuration:

Relevant Logs or Screenshots:

2

Do you want to backup the old indices in your OpenSearch cluster to S3? If so you can take a look at this document: Take and restore snapshots - OpenSearch documentation. Firstly, setup the env to register a s3 repository, and then call the snapshot API of OpenSearch to backup the data of the old indices to s3.

3

Hey thanks for the reply. I have enabled the snapshot and registered s3 repo. My requirement is to have take snapshot of the indices that is 50 days old and push it to s3 repo. I cannot see the option where we can specify the snapshot for 50 days old log.

4

You can use ISM to do that, define a policy that takes snapshot for the 50 days old indices, just like this:

{
    "id": "test33",
    "seqNo": 246143,
    "primaryTerm": 105,
    "policy": {
        "policy_id": "test33",
        "description": "A sample description of the policy",
        "last_updated_time": 1703056414992,
        "schema_version": 18,
        "error_notification": null,
        "default_state": "init",
        "states": [
            {
                "name": "init",
                "actions": [],
                "transitions": [
                    {
                        "state_name": "snapshot",
                        "conditions": {
                            "min_index_age": "50d"
                        }
                    }
                ]
            },
            {
                "name": "snapshot",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "snapshot": {
                            "repository": "example-repository",
                            "snapshot": "example-snapshot"
                        }
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "myindex-*"
                ],
                "priority": 1,
                "last_updated_time": 1703056386362
            }
        ]
    }
}

, take a look at the documentation of ISM: Policies - OpenSearch documentation