Upgrading Opensearch from 1.3.0 to 2.4.1 failed with unknown plugin "whoami"

Security
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

upgrade opensearch from 1.3.0 to 2.4.1

Describe the issue:

Upgrade failed with error:

"Will connect to localhost:9200 ... done",
        "ERR: An unexpected ResponseException occured: method [GET], host [https://localhost:9200], URI [/_plugins/_security/whoami], status line [HTTP/1.1 404 Not Found]",
        "Warnings: [[types removal] Specifying types in document get requests is deprecated, use the /{index}/_doc/{id} endpoint instead.]",
        "{\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [_plugins]\",\"index\":\"_plugins\",\"resource.id\":\"_plugins\",\"resource.type\":\"index_expression\",\"index_uuid\":\"_na_\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [_plugins]\",\"index\":\"_plugins\",\"resource.id\":\"_plugins\",\"resource.type\":\"index_expression\",\"index_uuid\":\"_na_\"},\"status\":404}",

Configuration:
default configurations

Relevant Logs or Screenshots:

I tried manually run the “securityadmin.sh” script and failed with same error in 2.4.1, but passes in 1.3.0 on the same master node:

bash /usr/local/opensearch/2.4.1/plugins/opensearch-security/tools/securityadmin.sh -cacert /etc/opensearch/client-certs/ca.pem -key /etc/opensearch/client-certs/key.p8 -cn graylog-es_elastic -cert /etc/opensearch/client-certs/cert.pem
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to localhost:9200 ... done
ERR: An unexpected ResponseException occured: method [GET], host [https://localhost:9200], URI [/_plugins/_security/whoami], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [_plugins]","index":"_plugins","resource.id":"_plugins","resource.type":"index_expression","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [_plugins]","index":"_plugins","resource.id":"_plugins","resource.type":"index_expression","index_uuid":"_na_"},"status":404}
Trace:
org.opensearch.client.ResponseException: method [GET], host [https://localhost:9200], URI [/_plugins/_security/whoami], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [_plugins]","index":"_plugins","resource.id":"_plugins","resource.type":"index_expression","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [_plugins]","index":"_plugins","resource.id":"_plugins","resource.type":"index_expression","index_uuid":"_na_"},"status":404}
	at org.opensearch.client.RestClient.convertResponse(RestClient.java:375)
	at org.opensearch.client.RestClient.performRequest(RestClient.java:345)
	at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
	at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:462)
	at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:159)

while in 1.3.0

bash /usr/local/opensearch/1.3.0/plugins/opensearch-security/tools/securityadmin.sh -cacert /etc/opensearch/client-certs/ca.pem -key /etc/opensearch/client-certs/.key.p8 -cn graylog-es_elastic -cert /etc/opensearch/client-certs/cert.pem -cd /usr/local/opensearch/1.3.0/plugins/opensearch-security/securityconfig
Security Admin v7
Will connect to localhost:9300 ... done
Connected as CN=graylog-es-elastic.service.auth
OpenSearch Version: 1.3.0
OpenSearch Security Version: 1.3.0.0
Contacting opensearch cluster 'graylog-es_elastic' and wait for YELLOW clusterstate ...
Clustername: graylog-es_elastic
Clusterstate: GREEN
Number of nodes: 3
Number of data nodes: 3
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/local/opensearch/1.3.0/plugins/opensearch-security/securityconfig/
Will update '_doc/config' with /usr/local/opensearch/1.3.0/plugins/opensearch-security/securityconfig/config.yml
   SUCC: Configuration for 'config' created or updated
Will update '_doc/roles' with /usr/local/opensearch/1.3.0/plugins/opensearch-security/securityconfig/roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '_doc/rolesmapping' with /usr/local/opensearch/1.3.0/plugins/opensearch-security/securityconfig/roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '_doc/internalusers' with /usr/local/opensearch/1.3.0/plugins/opensearch-security/securityconfig/internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '_doc/actiongroups' with /usr/local/opensearch/1.3.0/plugins/opensearch-security/securityconfig/action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '_doc/tenants' with /usr/local/opensearch/1.3.0/plugins/opensearch-security/securityconfig/tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Will update '_doc/nodesdn' with /usr/local/opensearch/1.3.0/plugins/opensearch-security/securityconfig/nodes_dn.yml
   SUCC: Configuration for 'nodesdn' created or updated
Will update '_doc/whitelist' with /usr/local/opensearch/1.3.0/plugins/opensearch-security/securityconfig/whitelist.yml
   SUCC: Configuration for 'whitelist' created or updated
Will update '_doc/audit' with /usr/local/opensearch/1.3.0/plugins/opensearch-security/securityconfig/audit.yml
   SUCC: Configuration for 'audit' created or updated
Done with success

The security plugin is healthy:

curl --key key.pem --cert cert.pem -H 'Content-Type: application/json' -X GET  https://localhost:9200/_plugins/_security/health
{"message":null,"mode":"strict","status":"UP"}

How do I make the opensearch index 2.4.1 compatible before trying to upgrade again?

2

@mojawina Did you try to run securityadmin.sh 1.3.0 and 2.4.1 against the same OpenSearch 1.3.0?

3

Hi @pablo Thanks for helping first of all. Yeah, the securityadmin.sh command I manually tested was run against the same cluster.

I was upgrading the cluster, and after 2.4.1 was installed and cluster tried to restart, this whoami call was one of the steps and failed.

Or were there some steps needed before the upgrade?

4

@mojawina Could you tell me how many nodes you have? Please describe the upgrade steps.

5

Hi @pablo , thank you for your help today. I was using Ansible to upgrade my sandbox 3 node cluster. It turned out for some reason Ansible failed to restart Opensearch during the upgrade, caused the whoami call failure. I had to manually restart Opensearch on the nodes using systemctl to update the running version to 2.4, then restarted Ansible tasks to finish the upgrade successfully.

Also turned out I could do a version check before the whoami call. And if the running version does not match with expected version, restart it.

Any other work around for this issue is welcome.