S3 DataSource Issue

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.13
Describe the issue:
opensearch.log shows below error
com.amazonaws.SdkClientException: Unable to execute HTTP request: emr-serverless.ap-southeast-2a.amazonaws.com
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.services.emrserverless.AWSEMRServerlessClient.doInvoke(AWSEMRServerlessClient.java:1174) ~[aws-java-sdk-emrserverless-1.12.651.jar:?]
at com.amazonaws.services.emrserverless.AWSEMRServerlessClient.invoke(AWSEMRServerlessClient.java:1141) ~[aws-java-sdk-emrserverless-1.12.651.jar:?]
at com.amazonaws.services.emrserverless.AWSEMRServerlessClient.invoke(AWSEMRServerlessClient.java:1130) ~[aws-java-sdk-emrserverless-1.12.651.jar:?]
at com.amazonaws.services.emrserverless.AWSEMRServerlessClient.executeStartJobRun(AWSEMRServerlessClient.java:848) ~[aws-java-sdk-emrserverless-1.12.651.jar:?]
at com.amazonaws.services.emrserverless.AWSEMRServerlessClient.startJobRun(AWSEMRServerlessClient.java:817) ~[aws-java-sdk-emrserverless-1.12.651.jar:?]
at org.opensearch.sql.spark.client.EmrServerlessClientImpl.lambda$startJobRun$0(EmrServerlessClientImpl.java:67) [spark-2.13.0.0.jar:?]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) [?:?]
at org.opensearch.sql.spark.client.EmrServerlessClientImpl.startJobRun(EmrServerlessClientImpl.java:63) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.spark.execution.session.InteractiveSession.open(InteractiveSession.java:61) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.spark.execution.session.SessionManager.createSession(SessionManager.java:43) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.spark.dispatcher.InteractiveQueryHandler.submit(InteractiveQueryHandler.java:94) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.spark.dispatcher.SparkQueryDispatcher.dispatch(SparkQueryDispatcher.java:101) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.spark.asyncquery.AsyncQueryExecutorServiceImpl.createAsyncQuery(AsyncQueryExecutorServiceImpl.java:43) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.spark.transport.TransportCreateAsyncQueryRequestAction.doExecute(TransportCreateAsyncQueryRequestAction.java:67) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.spark.transport.TransportCreateAsyncQueryRequestAction.doExecute(TransportCreateAsyncQueryRequestAction.java:28) [spark-2.13.0.0.jar:?]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:77) [opensearch-performance-analyzer-2.13.0.0.jar:2.13.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:395) [opensearch-security-2.13.0.0.jar:2.13.0.0]
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.13.0.0.jar:2.13.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.sql.spark.rest.RestAsyncQueryManagementAction.lambda$executePostRequest$1(RestAsyncQueryManagementAction.java:126) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.datasources.utils.Scheduler.lambda$withCurrentContext$0(Scheduler.java:30) [datasources-2.13.0.0.jar:?]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) [opensearch-2.13.0.jar:2.13.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
Caused by: java.net.UnknownHostException: emr-serverless.ap-southeast-2a.amazonaws.com

Configuration:

Relevant Logs or Screenshots:

2

After clicking on Search

image
image1905×956 64.2 KB

3

After clicking on See the full error

image

4

It lloks like name resolution issue from OpenSearch node.

5

Hi Pablo,
The url is changing with the changes in region tag below entry in opensearch.yml
plugins.query.executionengine.spark.config: ‘{“applicationId”:“*******”,“executionRoleARN”:“arn:aws:iam::********:role/EMRServerlessS3RuntimeRoleOPS”,“region”:“ap-southeast-2a”,“sparkSubmitParameters”:“–conf spark.dynamicAllocation.enabled=false”}’

if i change the region tag to ap-southeast-2b the url is changing to " [emr-serverless.ap-southeast-2b.amazonaws.com"
if i change region to blank url is changing to [emr-serverless…amazonaws.com]

I have followed url to create runtime role

thanks
Pravin

6

following below link to create S3glue connector

7

the above issue is resolved, wrong region.

having permissions issue now

2024-04-18T16:39:30,088][ERROR][o.o.s.s.c.EmrServerlessClientImpl] [awuavopsa01.citc.health.nsw.gov.au] Error while making start job request to emr:
com.amazonaws.services.emrserverless.model.AWSEMRServerlessException: User: arn:aws:sts:::assumed-role/OPS-Stack-OPSIamRole-bDADEKlimGO8/i-078cce831d8e8997a
is not authorized to perform: emr-serverless:TagResource on resource: arn:aws:emr-serverless:ap-southeast-2::/applications/********
(Service: AWSEMRServerless; Status Code: 403; Error Code: AccessDeniedException; Request ID: a871c952-afea-4e9a-98ba-58a7d804e6e3; Proxy: null)

8

assigned required policies