Migration es7.10.2 opensearch 2.4.0

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
elasticsearch 7.10.2
opendistro 1.13.1.0

to

Opensearch 2.4.0

Describe the issue:

My stack is composed of a set of docker containers managed by docker swarm,

I try to migrate es to opensearch using “full stack” method i.e. :

undeploying stack keeping only data volume for data and master nodes.
redeploying stack in opensearch
I use security ssl throught opendistro-security plugin and opensearch security plugin

A set of problems happens:

security index is not accessible from master (connect as null) : solution is to remove plugins.security.ssl.http.clientauth_mode: NONE from opensearch.yml

after this last operation security index becomes accessible from master (securityadmin.sh) but cluster is red.

on data nodes : first a mapping problem appear on .kibana_1 index:

java.lang.IllegalStateException: unable to upgrade the mappings for the index [[.kibana_1/t0tbokbhTq-INNPj19uujw]]
Likely root cause: MapperParsingException[No handler for type [flattened] declared on field [state]]

, removing this resolves the problem but cluster keep red.

on master node:
Trying to reload security config :

fails for each config file: config.yml , roles.yml …
Will update ‘/config’ with ./config.yml
FAIL: Configuration for ‘config’ failed because of java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-6 [ACTIVE]

on client node : always fails :

ERROR: [1] bootstrap checks failed

So is it possible to migrate by full stack down/up from elasticsearch 7.10.2 to opensearch 2.4.0 with ssl security activated ?

Configuration:
config elasticsearch
config opensearch

Relevant Logs or Screenshots:

2

Hey, if I understand correctly you seem to be trying to go from Elasticsearch 7.10.2 to OpenSearch 2.4.0. This is not supported currently however. To get to 2.4.0 you will need to first migrate Elasticsearch to OpenSearch 1.3.x (most current 1.3 version), then you should be able to upgrade to the 2.x line.

There were some breaking changes between the 1.x and 2.x line which is why you need to migrate this way.

4

That is a lot of logs. Glad we are getting a bit further. I am not the best with the security settings but I’ve moved this post to the security topic to see if someone else can have a look.

5

@comijac I’ve noticed that your cluster is in RED state.

Could you share the output of the following commands?

curl --insecure -u admin:admin -XGET https://<OpenSearch_node>:9200/_cluster/health
curl --insecure -u admin:admin -XGET https://<OpenSearch_node>:9200/_cat/nodes

curl --insecure -u admin:admin -XGET https://<OpenSearch_node>:9200/_cat/indices
curl --insecure -u admin:admin -XGET https://<OpenSearch_node>:9200/_cat/shards
6

As my stack is red : any request like

curl --insecure -u admin:admin -XGET https://<OpenSearch_node>:9200/_cluster/health

fails.

  • Trying 10.0.56.27:9200…
  • TCP_NODELAY set
  • connect to 10.0.56.27 port 9200 failed: Connection refused
  • Trying 10.0.56.28:9200…
  • TCP_NODELAY set
  • connect to 10.0.56.28 port 9200 failed: Connection refused
  • Trying 10.0.56.26:9200…
  • TCP_NODELAY set
  • connect to 10.0.56.26 port 9200 failed: Connection refused
  • Failed to connect to os-master.os-bj-test port 9200: Connection refused
  • Closing connection 0
    curl: (7) Failed to connect to os-master.os-bj-test port 9200: Connection refusedi

But I have found one mistake in my migration :
my node master opensearch after migration used not the same volume as node master elasticsearch before migration.

So preceeding error logs I paste have no interest and can be cancelled.

I corrected this point an now I have only the following error on each node master, client and data:

{“exception”:{“exception_class”:“java.lang.IllegalStateException”,“exception_message”:“transport not ready yet to handle incoming requests”,“stacktrace”:“java.lang.IllegalStateException: transport not ready yet to handle incoming requestsntat org.opensearch.transport.TransportService.onRequestReceived(TransportService.java:1110)ntat org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:182)ntat org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:127)ntat org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:109)ntat org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:759)ntat org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:170)ntat org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:145)ntat org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:110)ntat org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94)ntat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)ntat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)ntat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)ntat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)ntat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)ntat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)ntat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)ntat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)ntat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)ntat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)ntat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)ntat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371)ntat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234)ntat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283)ntat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)ntat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)ntat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)ntat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)ntat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)ntat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)ntat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)ntat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)ntat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)ntat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)ntat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)ntat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)ntat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)ntat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)ntat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)ntat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)ntat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)ntat java.base/java.lang.Thread.run(Unknown Source)n”},“@version”:1,“source_host”:“71ea127e358b”,“message”:“exception caught on transport layer [Netty4TcpChannel{localAddress=/10.0.56.28:9300, remoteAddress=/10.0.56.24:50128}], closing connection”,“thread_name”:“opensearch[os-master-3][transport_worker][T#3]”,“@timestamp”:“2023-01-18T11:29:43.033+01:00”,“level”:“WARN”,“logger_name”:“org.opensearch.transport.TcpTransport”}

trying to request security index I havr the following error:

Will connect to localhost:9300 … done
ERR: Cannot connect to OpenSearch. Please refer to opensearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{d7MnF96oTnem2WdJtpF7vA}{localhost}{127.0.0.1:9300}]]
at org.opensearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:381)
at org.opensearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:272)
at org.opensearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:79)
at org.opensearch.client.transport.TransportClient.doExecute(TransportClient.java:484)
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433)
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:419)
at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:524)
at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:157)

7

@comijac Could you share your opensearch.yml and docker-compose files?

Also share full securityadmin.sh command.

9

@comijac Did you use securityadmin.sh tool provided with OpenSearch 2.4.1?
I think you’re running an incorrect version of that tool. Since version 2.0 the client authentication through the transport layer (port 9300) has been deprecated.

The securityadmin.sh tool is using port 9200 to connect with the OpenSearch node now.

i.e.

image
image1214×575 149 KB

10

Thank for your reply

Finally I have migrate 2 clusters from es 7.10.2 to os 1.3.2 but it seems difficult and slow:

  • need to erase security index once : after elasticsearch cluster up then erase security index erase .kibana_1 index, migrate to opensearch and wait 10’ before cluster opensearch up (green)

Both cluster are green and ssl communication between nodes seems correct but :

  • for the first cluster securityadmin.sh requets are correct
  • for the second securityadmin.sh fails with : the precceding transport error

perhaps first embbed securityadmin.sh 1.3.2 the second embedded 2.4.1 , is it possible to displays this version without strting a securityadmin request ?

Let see above a set of logs noticed before cluster becomes up (10’ to 15’) :

A lot of logs in master nodes before the cluster switch from red to yellow then green: like

loop during 10’ on:
{“mdc”:{“user”:“admin”},“@version”:1,“source_host”:“b003811963b8”,“message”:“Authentication finally failed for admin from 10.2.170.70:51398”,“thread_name”:“opensearch[os-master-2][transport_worker][T#5]”,“@timestamp”:“2023-01-18T15:49:50.005+01:00”,“level”:“WARN”,“logger_name”:“org.opensearch.security.auth.BackendRegistry”}

and then logs like :
{“exception”:{“exception_class”:“java.lang.IllegalArgumentException”,“exception_message”:“unknown setting [index.lifecycle.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings”,“stacktrace”:"java.lang.IllegalArgumentException: unknown setting [index.lifecycle.name] please check that…

{“exception”:{“exception_class”:“org.opensearch.index.IndexNotFoundException”,“exception_message”:“no such index [.opendistro-ism-config]”,“stacktrace”:"[.opendistro-ism-config] IndexNotFoundException[no such index [.opendistro-ism-config]]ntat org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:1055)ntat org.op…

11

Hi @dtaivpp,

No additional update information appears in this resource when migrating from elasticsearch 7.x. Are you sure that we have to go through an opensearch 1.3 stage as you said to migrate from Elasticsearch 7.9 to a version like opensearch 2.4.0? Is there a resource on this?

12

It’s entrusting question, because while i trying upgrade from elasticsearch-7.9.1 to opensearch-1.3.14 i still have exception “java.lang.IllegalStateException: unable to upgrade the mappings for the index [[.kibana_1/TQCNBXXZTP6KdDwvcfTabQ]]”. Removing Kibana index don’t look like as good solution for my production clusters. ((