java.lang.IllegalArgumentException: index template - how critical?

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

opensearch-2.9.0-1.x86_64
opensearch-dashboards-2.9.0-1.x86_64

Describe the issue:
Hi guys.

A complete newbie thus asking - go easy on me.
I’ve just deployed on Centos 9 and see this:

uncaught exception in thread [main] java.lang.IllegalArgumentException: index template [ss4o_metrics_template] has index patterns [ss4o_metrics-*-*] matching patterns from existing templates [ss4o_metric_template] with patterns (ss4o_metric_template => [ss4o_metrics-*-*]) that have the same priority [1], multiple index templates may not match during index creation, please use a different priority at org.opensearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:558) at org.opensearch.cluster.metadata.MetadataIndexTemplateService$4.execute(MetadataIndexTemplateService.java:491) at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65) at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:874) at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424) at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295) at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206) at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204) at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242) at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) For complete error details, refer to the log at /var/log/opensearch/graylog.log

The set-up is pretty much vanilla-default, I only did tls-certs bits.
To see if system works I played with dashboards a bit and it seems to behave, but - how critical is that error? Is this an issue?

many thanks, L.

Configuration:

Relevant Logs or Screenshots:

1 Like

@lejeczek According to the error you have two index templates (ss4o_metric_template and ss4o_metrics_template) configured that are matching indices with ss4o_metrics-*-*pattern.

The index template defines the settings of the index, field mappings etc. It is used to override the default index configuration. Once a new index is created and the index name is matching the name of the index defined in the index template, the index will be created with all settings and mappings configured in that index template.

It might be confusing for OpenSearch when two or more index templates have the same index pattern.

Ok, but… any idea how that came to existence?
I’ll repeat that - pretty vanilla-default & that error was from the start one.

Also - just learning - In: Index mgmt → Templates | I see only tree templates, no ss4o_metrics_template

I wonder - this one server/node I started first then rpms updates came, afterwards I started a second server/node which “virtually” identical, is free from these errors.

Any suggestion on to “fix” are very much appreciated.

many thanks, L.

@lejeczek I’ve just deployed 2.9 and I have only these templates. Not sure how you got the extra one but it doesn’t come with the installation. At least not in the docker.

How did you deploy your cluster?

Yes, again, that is what I have - on the server which errors out - I see that which you showed - three templates but ss4o_metric_template & no ! ss4o_metrics_template
No docker, I did not mention it but said Centos, from dnf repos, I deployed it, played with it, next came rpm updates, only then, with updated package I set up second server which does not complain.
Perhaps there was an “issue” with previous rpm package(s) &| something gone bit wrong during (deployed) version update.

a) why dashboards do not show it, that template, yet journal logs do.
b) how to fix it, if possible.

It is possible to edit/amend template(s) or re-create/initialize them (or whatever the correct nomenclature is)?

There is nothing in this server - like I said I barely started - but I’d avoid destroying it and learn something by tampering with it.

@lejeczek What do you get when you run GET _cat/templates ?

How do I run it? (remember who you are dealing with, I confessed in my 1st message)

@lejeczek no worries

In OpenSearch Dashboards go to Dev Tools
image

Then you’ll access the Dev Tools console where you can type the API call

image

You’ll get the response in the right pane. Please share the output.

ss4o_metric_template [ss4o_metrics-*-*] 1 1 [] ss4o_trace_template [ss4o_traces-*-*] 1 1 [] tenant_template [.kibana_-*_*, .kibana_0*_*, .kibana_1*_*, .kibana_2*_*, .kibana_3*_*, .kibana_4*_*, .kibana_5*_*, .kibana_6*_*, .kibana_7*_*, .kibana_8*_*, .kibana_9*_*] 2147483647 []

Is the issue just the template name?
If so - if I use what view gets me, in JSON and then - deleting the temple in-between - then use that with
PUT _index_template/ss4o_metrics_template

would that be okey?

Ok. I did that. That error is now no more but this:

uncaught exception in thread [main] java.lang.IllegalArgumentException: index template [ss4o_traces_template] has index patterns [ss4o_traces-*-*] matching patterns from existing templates [ss4o_trace_template] with patterns (ss4o_trace_template => [ss4o_traces-*-*]) that have the same priority [1], multiple index templates may not match during index creation, please use a different priority

If devel read this - it feels like something breaks with/during rpms updates - on deployed, up & running server, my second box which got updates before its first run did not “suffer” from this.
Am I guessing - sure yes.

Ok, now… journal-log error-free the server is.
But will what I did not cause some harm, unwanted misbehaviour/effects - any expert/devel can comment?

thanks, L.

@lejeczek Please share responses from the below.

GET _index_template/ss4o_traces_template
GET _index_template/ss4o_trace_template

I don’t think that the RPM update would change anything in the configuration of the OpenSearch Dashboards or OpenSearch.

Check my last message.
I do think that somewhere there was(is) the problem - rpms/updates, who else would change templates names, cut s short but devel - all that happen to the server I described already. Nothing else was done to, again, barely logged into dashboards once.

@lejeczek I know what happened. You must have OpenSearch 2.8 or older first on that node. Then you’ve upgraded to 2.9.

This is the output from OpenSearch 2.8

This is the output from OpenSearch 2.9

As you can see there is a difference in the template’s name.
When you upgraded from version 2.8 (I assume) to 2.9, the data (indices) including index templates remained. The new version presented extra index templates which were pointing to the same index pattern and had the same priority ID.

I would say this is a bug. Please report it to OpenSearch GitHub.
Once you report the bug, please share the link in this thread.

1 Like

Gee… man barely put my first foot into it first time - happens way too often.
Filed.
thanks, L.
ps. what it did to templates - will it be ok?

@lejeczek That PUT API call has cleared the configuration of that index template. That’s why you don’t see errors any more.

I had a look at both index templates and they look the same. I think you should be ok.