Policy ISM - Rollover

OpenSearch
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 2.5.0

Hi,

I’m opening this new topic because I’m facing a problem, I apologize because I don’t have much knowledge in OpenSearch management.

My Scenario:

1. I have logs coming from Fluent Bit, I have the following configuration:

[OUTPUT]
    Name                opensearch
    Match               application.*
    AWS_Region          ${AWS_REGION}
    Host                <HOSTNAME OPENSEARCH>
    Port                443
    Logstash_Format     True
    Logstash_Prefix     eks
    Type                _doc
    tls                 On
    tls.verify          off
    HTTP_User           ${FLUENBIT_USER}
    HTTP_Passwd         ${FLUENBIT_PASS}
    Suppress_Type_Name  On

With these parameters Logstash_Format and Logstash_Prefix it generates the following index in OpenSearch:

eks-<DATE>
eks-2023.04.03

2. An ISM policy was created: Below is the excerpt:

{
  "name": "rollover",
  "actions": [
    {
      "rollover": {
        "min_doc_count": 1
      }
    }
  ],
          "transitions": [
              {
                  "state_name": "hot",
                  "conditions": {
                      "min_rollover_age": "10m"
                  }
              }
          ]
}

3. Created the following template:

{
  "index_patterns": ["eks*"],
  "template": {
   "settings": {
    "plugins.index_state_management.rollover_alias": "eks"
   }
 }
}

4. Created the following alias:

{
  "aliases": {
    "eksalias": {
      "is_write_index": true
    }
  }
}

Expected scenario: The objective was to generate a new file with the date every day, and during the day to perform the rollover.

Example: eks-2023.04.03-00001, eks-2023.04.03-00002…

However, when performing these steps, I get the following error:

“message”: “Missing alias or not the write index when rollover [index=eks-2023.04.03]”

Could someone please help?

Thank you very much

2

Hi @diogosilva in the index template you have mentioned rollover alias as “eks” but while creating the index you use “eksalias”. please replace those. Use below api call
PUT eks-000001
{
“aliases”: {
“eks”: {
“is_write_index”: true
}
}
}
try it once

3

@pratikw

Thank you very much for your help, but then the name of the index would not be that …

The index comes from Fluent Bit with the following name eks-2023.04.04…but every day it will generate a new name, according to the date.

image
image731×82 6.17 KB

But during the day, I want to rollover these indexes, for example eks-2023.04.04…it would look something like this eks-2023.04.04-0001, eks-2023.04.04-0002, and so on.

Both the alias that I had indicated above and the one you indicated were created, and both had problems when performing the rollover.

image
image845×273 11.7 KB

It was tried to create an alias with the following method but it shows as invalid.

PUT eks-*
{
"aliases": {
"eks": {
"is_write_index": true
}
}
}
4

If your bit is already rolling over the indexes then why you are applying rollover on those index… instead of that you can setup ism policy by disabling the rollover action…and move them from one phase to another(I mean from hot to warm to cold like this)

5

And still of you want a control over rollover maybe you can setup your own template in that beats and overwrite the default one if possible.

6

Yes, Fluent Bit already does it, but during the day I want the rollover due to the amount of logs.

7

AFAIK, the index naming in Fluent Bit is controlled by Logstash_Format and Logstash_Prefix (which you have) and the Logstash_DateFormat option controls how the index suffix looks like.

I think that you’ll want to point Fluent Bit to your alias, which implies removing the settings mentioned above (the default Logstash_Format is Off, at least in the latest 2.0 version) and set Index to your alias name instead. It looks like that’s eks, if you followed what @pratikw suggested in the first reply. Something like this won’t work:

PUT eks-*
{
"aliases": {
"eks": {
"is_write_index": true
}
}
}

Because you’re trying to put an index name with a wildcard in its name. Try with PUT eks-000001, like @pratikw suggested.