Why the _cat/nodes API indicates differently for "node.roles" and "node.role"

[latituder]

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.18.0

Describe the issue:
I have issue in run the Transform jobs in opensaerch, when I try to search logs I only see very less logs from the data role nodes, but not see my transform node log indicate anyting about the transform error.

So I try to troubleshoot the transform role node, and call _cat/nodes API with ‘v’ parameter.
to my surprise, I see there are 2 columes in the response body relate to node role:
in node.roles, i can see the disired node has transform role configured.
however in node.role node it shows differently, like the node is not really taking the transfrom role.
(see screenshot, it only show irs, means ingest, remote_cluster_client and search.
i expect to see a “t” in the colume, but it is not there.

i checked the pod log, only 2 info level messages:
[2025-01-02T03:58:33,638][INFO ][o.o.n.Node ] [ssdl-app-logging-opensearch-i-s-t-r-0] node name [ssdl-app-logging-opensearch-i-s-t-r-0], node ID [kQnyuv98ST20XKXmW5198A], cluster name [ssdl-app-logging-opensearch], roles [ingest, remote_cluster_client, transform, search]
[2025-01-02T03:58:42,237][INFO ][o.o.d.PeerFinder ] [ssdl-app-logging-opensearch-i-s-t-r-0] setting findPeersInterval to [1s] as node commission status = [true] for local node [{ssdl-app-logging-opensearch-i-s-t-r-0}{kQnyuv98ST20XKXmW5198A}{qZ45O-QqQei8bid866UFfQ}{100.104.15.84}{100.104.15.84:9300}{irstransform}{shard_indexing_pressure_enabled=true}]

so my questions are:

1. why the transform job mesasge appears in data node in my cluster?
2. why the transform role node doesn’t has ‘t’ mark in _cat/nodes response body?

Configuration:
p.s: my cluster was implemented by opensearch-operator.
a part of the OS CR is:
- component: i-s-t-r
… …
roles:
- “ingest”
- “search”
- “transform”
- “remote_cluster_client”

Relevant Logs or Screenshots:

[gaobinlong]

By my test, node role can be arbitrary string, and there’s no role named transform in OpenSearch, so the node.role column in the response of _cat/nodes doesn’t show the abbreviation of that role.

[latituder]

this is a surprise to me.
from the opensearch repo, I haven’t find any node role difinations.
however, from the opensearch-operator repo, I see a difination of the available node roles:

opensearch-project/opensearch-k8s-operator/blob/2e78967d3d5868d719442574f5d26ee8fae4a17a/opensearch-operator/pkg/builders/cluster.go#L60

and I see the “transform” role is defined there.

if the “transform” is not a valid node role, what name it shall be?

[latituder]

OK, I think you are right.
the “transform” is a node role for elasticsearch:
Transforms settings in Elasticsearch | Elasticsearch Guide [8.17] | Elastic

however, it is not a valid role for opensearch:
Configuration and system settings - OpenSearch Documentation

[latituder]

may I ask a similar question:
in opensearch doc, it mentioned node role “ml” for machine learning nodes.
I tried to add the “ml” role to the cluster, but it also not actually proceed by the node.
is this mean the “ml” role actually also not exist?

[gaobinlong]

ml node role exists in OpenSearch, how did you find the ml task is not proceeded on the ml node?

[jakabasej5]

@gaobinlong

By my test, node role can be arbitrary string, and there’s no role named transform in OpenSearch, so the node.role coltelli giapponesi da cucina column in the response of _cat/nodes doesn’t show the abbreviation of that role.

this is perfect!