Upgrade from Elasticsearch Basic to OpenSearch

Hi,
I am trying to upgrade from a Elasticsearch Basic cluster (i.e. not OSS) to OpenSearch.

Looks like Upgrade from Elasticsearch OSS to OpenSearch - OpenSearch documentation is only applicable for upgrade from Elasticsearch OSS. I tried Upgrade from Elasticsearch OSS to OpenSearch - OpenSearch documentation to perform rolling upgrade for a Elasticsearch 7.8 Basic cluster. The first node that i upgraded is not able to join the existing cluster with the following error:

Caused by: org.opensearch.transport.RemoteTransportException: [***][***:***][internal:cluster/coordination/join/validate]
Caused by: java.lang.IllegalArgumentException: Unknown NamedWriteable [org.opensearch.cluster.metadata.Metadata$Custom][licenses]

Any suggestions?

  1. Is it possible to do a 2 step rolling upgrade from Elasticsearch Basic to Elasticsearch OSS and then from Elasticsearch OSS to OpenSearch?
  2. Will i be able to upgrade Elasticsearch Basic to OpenSearch directly by upgrading all the nodes at once, i.e. stop all nodes, upgrade all nodes to OpenSearch, start all nodes?

Option 2 worked for me!

  1. Will i be able to upgrade Elasticsearch Basic to OpenSearch directly by upgrading all the nodes at once, i.e. stop all nodes, upgrade all nodes to OpenSearch, start all nodes?
3 Likes

If anyone runs into this, theoretically you should be able to do a rolling upgrade, but that error tells me there might be a failing check on the cluster side. Is there more to it in the master node logs? Try setting compatibility.override_main_response_version before starting the node, too, via opensearch.yml, see if there’s any difference.

Either way, I would open an issue so we can at least track it in GitHub - opensearch-project/OpenSearch: 🔎 Open source distributed and RESTful search engine.

1 Like

@dblock I tried adding compatibility.override_main_response_version and get the same error as shown below on the first node that is upgraded to OpenSearch

2021-10-27T17:53:50,858 main         [W] org.ope.clu.met.Metadata            - [UID=] - Skipping unknown custom object with type index_lifecycle
2021-10-27T17:53:50,858 main         [W] org.ope.clu.met.Metadata            - [UID=] - Skipping unknown custom object with type licenses
...
2021-10-27T17:57:31,318 eneric][T#4] [W] org.ope.clu.coo.JoinHelper          - [UID=] - last failed join attempt was 876ms ago, failed to join {example01}{apj5eEI-TEq15shrkw1fXQ}{KwQoWYYGRNS8I5wJt7Shaw}{example01.example.com}{*.92:*}{dm}{xpack.installed=true, transform.node=false} with JoinRequest{sourceNode={example02}{Dhps48KRQUWWY-b-vqfmEg}{Cdq8kodDScGEyqJTvsykxQ}{example02.example.com}{*.93:*}{dm}, minimumTerm=1, optionalJoin=Optional.empty}
org.opensearch.transport.RemoteTransportException: [example01][*.92:*][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
        at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:512) ~[?:?]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[?:?]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1173) ~[?:?]
        at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.opensearch.transport.RemoteTransportException: [example02][*.93:*][internal:cluster/coordination/join/validate]
Caused by: java.lang.IllegalArgumentException: Unknown NamedWriteable [org.opensearch.cluster.metadata.Metadata$Custom][licenses]
        at org.opensearch.common.io.stream.NamedWriteableRegistry.getReader(NamedWriteableRegistry.java:125) ~[opensearch-1.1.0.jar:1.1.0]
        at org.opensearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:58) ~[opensearch-1.1.0.jar:1.1.0]
        at org.opensearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:52) ~[opensearch-1.1.0.jar:1.1.0]
        at org.opensearch.cluster.metadata.Metadata.readFrom(Metadata.java:1029) ~[opensearch-1.1.0.jar:1.1.0]
        at org.opensearch.cluster.ClusterState.readFrom(ClusterState.java:718) ~[opensearch-1.1.0.jar:1.1.0]
        at org.opensearch.cluster.coordination.ValidateJoinRequest.<init>(ValidateJoinRequest.java:46) ~[opensearch-1.1.0.jar:1.1.0]
        at org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:72) ~[opensearch-1.1.0.jar:1.1.0]
        at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:208) ~[opensearch-1.1.0.jar:1.1.0]
        at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:120) ~[opensearch-1.1.0.jar:1.1.0]
        at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:102) ~[opensearch-1.1.0.jar:1.1.0]
        at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:713) ~[opensearch-1.1.0.jar:1.1.0]
        at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:155) ~[opensearch-1.1.0.jar:1.1.0]
        at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:130) ~[opensearch-1.1.0.jar:1.1.0]
        at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:95) ~[opensearch-1.1.0.jar:1.1.0]
        at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:87) ~[?:?]

I think you should open a bug in OpenSearch repo and we’ll take a look.

@dblock create a bug for this [BUG] Rolling Upgrade from Elasticsearch Basic to OpenSearch fails in cluster validation · Issue #1502 · opensearch-project/OpenSearch · GitHub. thanks!

Hey @ronniepg - I saw this issue got closed. The upgrade path from basic license ES to OpenSearch isn’t supported.

It brings in a ton of complexities that are really hard to manage (including similar, but incompatible functionality), not to mention that ES Basic is checking for licenses on OpenSearch node - there is no way to control for that.

That being said, if someone was to contribute a working migration path I don’t think it would be turned down, but it would be under a lot of scrutiny to make sure any contributed code was not lifted from a proprietary code base.