Help needed for creating a template for merging & deleting old indices

[stmx38]

Hello @lehner.angelica,
At the moment we still use Elasticsearch Curator for index cleanup and started to consider to move to the Index Management. This post can be as a short guide for us.

Per documentation, in order to just cleanup old indices by wildcard we should:

1. Define a Policy which should delete old indices: Kibana --> Index management Kibana --> Create policy
   Policy ID: vpn-log-test-cleanup
   Define policy:

   {
     "policy": {
       "description": "Policy that deletes indices 'vpn-log-test' older than 30 days",
       "default_state": "open",
       "schema_version": 1,
       "states": [
         {
           "name": "open",
           "actions": [],
           "transitions": [
             {
               "state_name": "delete",
               "conditions": {
                 "min_index_age": "30d"
               }
             }
           ]
         },
         {
           "name": "delete",
           "actions": [
             {
               "delete": {}
             }
           ]
         }
       ]
     }
   }
   

2. Create an Index template, to be able to attach policy to multiple indices by wildcard and attach created policy to it. It can be done via Dev Tools/Console or cURL:
   Dev Tools
   Create Index template

   PUT _template/vpn-log-test
   {
     "index_patterns": [
       "vpn-log-test-*"
     ],
     "settings": {
       "opendistro.index_state_management.policy_id": "vpn-log-test-cleanup"
     }
   }
   

   Check the result

   GET _template/vpn-log-test
   

   cURL
   Create Index template

   curl -X PUT http://localhost:9200/_template/vpn-log-test -H 'Content-Type: application/json' -d'
   {
     "index_patterns": [
       "vpn-log-test-*"
     ],
     "settings": {
       "opendistro.index_state_management.policy_id": "vpn-log-test-cleanup"
     }
   }'
   

   Get created template

   curl http://localhost:9200/_template/vpn-log-test?pretty
   

Testing

Note: Index management policy will be attached to the indice in the moment of its creation because attachment is described in the Index template. It means that the policy will be attached to the newly index only.

1. Create a new Index

   # Variables
   elasticsearch_url=http://localhost:9200
   date=$(date +%Y-%m-%d)
   index_name=vpn-log-test-$date
   index_type=default

   users="Alice Bob"
   error="VPN connection failed"

   # Log to the Elasticsearch
   for user in $users; do
     time=$(date +%Y-%m-%d'T'%H:%M:%S.%3N)

     curl -H "Content-Type: application/json" \
        -XPOST "$elasticsearch_url/$index_name/$index_type" \
        -d "{\"Time\":\"$time\", \"User\":\"$user\", \"Error\":\"$error\"}"
        sleep 2
   done

2. Check if policy was attached

We see that only the indice created today (righ now) is ‘Managed by Policy’, as it was described in the note above.

3. For testing purposed we changed conditions to the 1m

        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "1m"
            }
          }
        ]

4. We see that policy status was changed to the Initializing

5. After 5 minutes of waiting (Index management scheduler running period) we see that policy change to the Running

6. Finaly, we see that indice dissapeared from the list as it was deleted by Index management