@dblock I tried adding compatibility.override_main_response_version
and get the same error as shown below on the first node that is upgraded to OpenSearch
2021-10-27T17:53:50,858 main [W] org.ope.clu.met.Metadata - [UID=] - Skipping unknown custom object with type index_lifecycle
2021-10-27T17:53:50,858 main [W] org.ope.clu.met.Metadata - [UID=] - Skipping unknown custom object with type licenses
...
2021-10-27T17:57:31,318 eneric][T#4] [W] org.ope.clu.coo.JoinHelper - [UID=] - last failed join attempt was 876ms ago, failed to join {example01}{apj5eEI-TEq15shrkw1fXQ}{KwQoWYYGRNS8I5wJt7Shaw}{example01.example.com}{*.92:*}{dm}{xpack.installed=true, transform.node=false} with JoinRequest{sourceNode={example02}{Dhps48KRQUWWY-b-vqfmEg}{Cdq8kodDScGEyqJTvsykxQ}{example02.example.com}{*.93:*}{dm}, minimumTerm=1, optionalJoin=Optional.empty}
org.opensearch.transport.RemoteTransportException: [example01][*.92:*][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:512) ~[?:?]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[?:?]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1173) ~[?:?]
at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:235) ~[?:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.opensearch.transport.RemoteTransportException: [example02][*.93:*][internal:cluster/coordination/join/validate]
Caused by: java.lang.IllegalArgumentException: Unknown NamedWriteable [org.opensearch.cluster.metadata.Metadata$Custom][licenses]
at org.opensearch.common.io.stream.NamedWriteableRegistry.getReader(NamedWriteableRegistry.java:125) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:58) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:52) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.cluster.metadata.Metadata.readFrom(Metadata.java:1029) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.cluster.ClusterState.readFrom(ClusterState.java:718) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.cluster.coordination.ValidateJoinRequest.<init>(ValidateJoinRequest.java:46) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:72) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:208) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:120) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:102) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:713) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:155) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:130) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:95) ~[opensearch-1.1.0.jar:1.1.0]
at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:87) ~[?:?]