Index template not applied to new top_queries- indexes

OpenSearch
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

  • Opensearch 2.19.0

Describe the issue:
After the update to opensearch 2.19.0 I noticed that my one-node opensearch “cluster” is running in state “yellow” every morning.

Apparently this is caused by the “Top N queries”-feature and the resulting top_queries-YYYY.MM.DD-NNNNN indices which are created every night (with "number_of_replicas": "1").

I’ve adapted my index templates to also apply to the top_queries-* indexes but apparently this has no effect as new top_queries- are still created with "number_of_replicas": "1".

{
  "index_patterns": [
    ".opendistro-*",
    ".opensearch-*",
    "top_queries-*"
  ],
  "template": {
    "settings": {
      "index.number_of_shards": "1",
      "index.number_of_replicas": "0"
    }
  },
  [...]
  "name": "system-indices"
}

Is this a bug or misconfiguration on my end? Any ideas on how to permanently fix this?
Configuration:

Relevant Logs or Screenshots:

2

Hi @mauno,

Would you mind sharing the output of the following:

GET _cat/templates

best,
mj

3

@Mantas not at all. Thanks for takin a look. As far as I can see there is no other index template matching the top_queries-* pattern…

ecs-logstash        [ecs-logstash-*] 1
system-indices      [.opendistro-*, .opensearch-*, top_queries-*] 1 []
datastream-template [buk-logs-*] 0 []
tenant_template     [.kibana_-*_*, .kibana_0*_*, .kibana_1*_*, .kibana_2*_*, .kibana_3*_*, .kibana_4*_*, .kibana_5*_*, .kibana_6*_*, .kibana_7*_*, .kibana_8*_*, .kibana_9*_*] 2147483647  []
4

yep, it looks like you are correct. Just to have a full picture could you also share:

GET _index_template/system-indices
5

Sure…

{
  "index_templates": [
    {
      "name": "system-indices",
      "index_template": {
        "index_patterns": [
          ".opendistro-*",
          ".opensearch-*",
          "top_queries-*"
        ],
        "template": {
          "settings": {
            "index": {
              "number_of_shards": "1",
              "number_of_replicas": "0"
            }
          }
        },
        "composed_of": [],
        "priority": 1,
        "_meta": {
          "flow": "simple"
        }
      }
    }
  ]
}
6

Hi @mauno,

Could you increase the priority for testing purposes to, let’s say, 1000 (to make sure the template is applied, tho I don’t see a good reason why it shouldn’t be…)?

best,
mj

7

Sorry for my late reply.
Setting the priority of the system-indices-template to 1000 had no effect on the issue.

8

Hi there.
Unfortunately, I discovered same problem in upgrading from 2.15 to 2.19 .
From my perspective, this looks like a bug and specially it hitting one multiroles nodes.
@Mantas have you please any additional reference to the issue/bug tracking? If not, no problem, I can create one, just let me know.
Best regards!

1 Like
9

@LHozzan, Would you mind creating and documenting your findings? Please share it here.

Thanks,
mj

1 Like
10

It seems, that bug wasnt reported, so I made one on [BUG] Indices `top_queries-*` are permanently in YELLOW state · Issue #256 · opensearch-project/query-insights · GitHub .
I suggesting to continue with investigation and add information to the issue.

2 Likes
11

Thanks @mauno @LHozzan for pointing out the issue. Primary and replica count is defined in the top-queries-* index create request, so index templates of any priority will not override these values. Source code will need to be changed for this. We will post updates on the GitHub issue soon.

2 Likes