Trying to upgrade from elastic search. " Received message from unsupported version"

[Unquote8740]

Hello,

Im working on trying to upgrade an existing cluster (es 7.0.0) to opensearch, by using a rolling upgrade process, i have a new OpenSearch node built running 1.3.1, and when i start the node up, im getting and “Received message from unsupported version” message.

I’m sure I’m missing something obvious here.

[2022-05-05T11:19:48,867][INFO ][o.o.c.c.JoinHelper       ] [os-staging-01.duh-stage] failed to join {ip-10-49-1-181.us-west-2.compute.internal}{SbdvE2nbRC-Sa34AAsJrYA}{2A0qVPcGR3KzLNxZeSD9Fw}{10.49.1.181}{10.49.1.181:9300}{dim}{ml.machine_memory=33072087040, rack=us-west-2a, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={os-staging-01.duh-stage}{ebEUk3R2RVmRnxKOtQxEew}{nHIFHs4HR7GpbT5U9eeF_w}{10.49.1.136}{10.49.1.136:9300}{dim}{rack=us-west-2a, shard_indexing_pressure_enabled=true}, minimumTerm=4, optionalJoin=Optional.empty}
org.opensearch.transport.RemoteTransportException: [ip-10-49-1-181.us-west-2.compute.internal][10.49.1.181:9300][internal:cluster/coordination/join]
Caused by: org.opensearch.transport.ConnectTransportException: [os-staging-01.duh-stage][10.49.1.136:9300] general node connection failure
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener$1.onFailure(TcpTransport.java:1284) ~[?:?]
	at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:138) ~[?:?]
	at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:115) ~[?:?]
	at org.elasticsearch.transport.TcpTransport$1.doRun(TcpTransport.java:975) ~[?:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[?:?]
	at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:967) ~[?:?]
	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:942) ~[?:?]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[?:?]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.IllegalStateException: Received message from unsupported version: [35.24.79-beta2] minimal compatible version is: [6.7.0]
	at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:139) ~[?:?]
	at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:115) ~[?:?]
	at org.elasticsearch.transport.TcpTransport$1.doRun(TcpTransport.java:975) ~[?:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[?:?]
	at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:967) ~[?:?]
	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:942) ~[?:?]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[?:?]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
	at java.lang.Thread.run(Thread.java:835) ~[?:?]
[2022-05-05T11:19:48,872][INFO ][o.o.c.c.JoinHelper       ] [os-staging-01.duh-stage] failed to join {ip-10-49-1-181.us-west-2.compute.internal}{SbdvE2nbRC-Sa34AAsJrYA}{2A0qVPcGR3KzLNxZeSD9Fw}{10.49.1.181}{10.49.1.181:9300}{dim}{ml.machine_memory=33072087040, rack=us-west-2a, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={os-staging-01.duh-stage}{ebEUk3R2RVmRnxKOtQxEew}{nHIFHs4HR7GpbT5U9eeF_w}{10.49.1.136}{10.49.1.136:9300}{dim}{rack=us-west-2a, shard_indexing_pressure_enabled=true}, minimumTerm=4, optionalJoin=Optional.empty}
org.opensearch.transport.RemoteTransportException: [ip-10-49-1-181.us-west-2.compute.internal][10.49.1.181:9300][internal:cluster/coordination/join]
Caused by: org.opensearch.transport.ConnectTransportException: [os-staging-01.duh-stage][10.49.1.136:9300] general node connection failure
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener$1.onFailure(TcpTransport.java:1284) ~[?:?]
	at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:138) ~[?:?]
	at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:115) ~[?:?]
	at org.elasticsearch.transport.TcpTransport$1.doRun(TcpTransport.java:975) ~[?:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[?:?]
	at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:967) ~[?:?]
	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:942) ~[?:?]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[?:?]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
	at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.IllegalStateException: Received message from unsupported version: [35.24.79-beta2] minimal compatible version is: [6.7.0]
	at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:139) ~[?:?]
	at org.elasticsearch.transport.TransportHandshaker$HandshakeResponseHandler.handleResponse(TransportHandshaker.java:115) ~[?:?]
	at org.elasticsearch.transport.TcpTransport$1.doRun(TcpTransport.java:975) ~[?:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[?:?]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[?:?]
	at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:967) ~[?:?]
	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:942) ~[?:?]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:753) ~[?:?]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:53) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[?:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[?:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
	at java.lang.Thread.run(Thread.java:835) ~[?:?]

[Unquote8740]

have also found [BUG]Version Compatibility failure when OpenSearch node tries to join ElasticSearch node[7.10 version] · Issue #669 · opensearch-project/OpenSearch · GitHub that seems related or similar.

[Unquote8740]

same issue with 1.3.2

[Unquote8740]

this also seems related.

Upgrade from Elasticsearch Basic to OpenSearch

@dblock I tried adding compatibility.override_main_response_version and get the same error as shown below on the first node that is upgraded to OpenSearch 2021-10-27T17:53:50,858 main [W] org.ope.clu.met.Metadata - [UID=] - Skipping unknown custom object with type index_lifecycle 2021-10-27T17:53:50,858 main [W] org.ope.clu.met.Metadata - [UID=] - Skipping unknown custom object with type licenses ... 2021-10-27T17:57:31,318 eneric][T#4] [W] org.ope.clu.coo.…