_reindex null_pointer_exception

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 1.3 (AWS)

Describe the issue:
I have 2 OpenSearch Services Domaines in the same region in a VPC, the version of these domains is 1.3
I want to copy an index from one domain to another with _reindex, I followed the configuration of this link Migrating Amazon OpenSearch Service indexes using remote reindex - Amazon OpenSearch Service
Reindex data with the AWS Management Console

When I run the command in the Dashboards

POST _reindex
{
  "source": {
    "remote": {
      "host": "https://aos-xxx.amazonaws.com:443"
    },
    "index": "my_index"
  },
  "dest": {
    "index": "my_index"
  }
}

I got the error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "null_pointer_exception",
        "reason" : null
      }
    ],
    "type" : "null_pointer_exception",
    "reason" : null
  },
  "status" : 500
}

In my_index of remote domain I didn’t configure

    "_source": {
      "enabled": true
    }

So I suppose the default value is “enabled”: true

What’s the problem of _reindex?

Configuration:

Relevant Logs or Screenshots:

2

@Garance Could you run this API again and follow the logs of the OpenSearch and OpenSearch Dashboards? Please share all the errors and warnings.

3

@pablo here are logs of local domain OpenSearch from CloudWatch:

[2023-10-18T14:24:29,486][WARN ][r.suppressed             ] [7a191c651e3504ea060c95c7d3ec18b6] path: __PATH__ params: {pretty=true}
java.lang.NullPointerException
	at __PATH__(StringBuilder.java:128)
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
	at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:133)
	at org.apache.http.impl.nio.client.MainClientExec.prepareRequest(MainClientExec.java:520)
	at org.apache.http.impl.nio.client.MainClientExec.prepare(MainClientExec.java:146)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:128)
	at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
	at org.opensearch.client.RestClient.lambda$performRequestAsync$1(RestClient.java:390)
	at org.opensearch.client.Cancellable.runIfNotCancelled(Cancellable.java:93)
	at org.opensearch.client.RestClient.performRequestAsync(RestClient.java:388)
	at org.opensearch.client.RestClient.performRequestAsync(RestClient.java:375)
	at org.opensearch.index.reindex.remote.RemoteScrollableHitSource.execute(RemoteScrollableHitSource.java:190)
	at org.opensearch.index.reindex.remote.RemoteScrollableHitSource.lookupRemoteVersion(RemoteScrollableHitSource.java:112)
	at org.opensearch.index.reindex.remote.RemoteScrollableHitSource.doStart(RemoteScrollableHitSource.java:101)
	at org.opensearch.index.reindex.ScrollableHitSource.start(ScrollableHitSource.java:97)
	at org.opensearch.index.reindex.AbstractAsyncBulkByScrollAction.start(AbstractAsyncBulkByScrollAction.java:306)
	at org.opensearch.index.reindex.Reindexer.lambda$execute$0(Reindexer.java:156)
	at org.opensearch.index.reindex.BulkByScrollParallelizationHelper.executeSlicedAction(BulkByScrollParallelizationHelper.java:120)
	at org.opensearch.index.reindex.Reindexer.execute(Reindexer.java:136)
	at org.opensearch.index.reindex.TransportReindexAction$1.onResponse(TransportReindexAction.java:95)
	at org.opensearch.index.reindex.TransportReindexAction$1.onResponse(TransportReindexAction.java:92)
	at org.opensearch.index.reindex.BulkByScrollParallelizationHelper.initTaskState(BulkByScrollParallelizationHelper.java:158)
	at org.opensearch.index.reindex.Reindexer.initTask(Reindexer.java:131)
	at org.opensearch.index.reindex.TransportReindexAction.doExecute(TransportReindexAction.java:92)
	at org.opensearch.index.reindex.TransportReindexAction.doExecute(TransportReindexAction.java:58)
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:209)
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:120)
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:207)
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99)
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:207)
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:184)
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:103)
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:108)
	at org.opensearch.index.reindex.AbstractBaseReindexRestHandler.lambda$doPrepareRequest$0(AbstractBaseReindexRestHandler.java:73)
	at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:128)
	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:306)
	at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:392)
	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:235)
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
__AMAZON_INTERNAL__
	at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
__AMAZON_INTERNAL__
	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
__AMAZON_INTERNAL__
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:370)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:957)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at __PATH__(Thread.java:829)
4

@Garance Is that the only warning/error you get from OS and OSD?

https://aos-xxx.amazonaws.com:443

You’re using a port 443. Is that a reverse proxy to port 9200 of OS node?

Also, have you tried to connect with username and password to the remote idex?

5

@pablo the port 443 is in the article Migrating Amazon OpenSearch Service indexes using remote reindex - Amazon OpenSearch Service
I disabled the authentification of OpenSearch.
When I tried https://aos-xxx.amazonaws.com:9200 I got the same error.

6

After experiencing the same, what worked for me was adding "external": true to the remote object, i.e.:

POST _reindex
{
  "source": {
    "remote": {
      "host": "https://remote-host.es.amazonaws.com:443",
      "region": "us-east-1",
      "socket_timeout": "60m",
      "external": true
    },
    "size": 100,
    "index": "remote_index"
  },
  "dest": {
    "index": "local_index"
  }
}