S3 DataSource Issue

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.13
Describe the issue:
opensearch.log shows below error
com.amazonaws.SdkClientException: Unable to execute HTTP request: emr-serverless.ap-southeast-2a.amazonaws.com
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[aws-java-sdk-core-1.12.651.jar:?]
at com.amazonaws.services.emrserverless.AWSEMRServerlessClient.doInvoke(AWSEMRServerlessClient.java:1174) ~[aws-java-sdk-emrserverless-1.12.651.jar:?]
at com.amazonaws.services.emrserverless.AWSEMRServerlessClient.invoke(AWSEMRServerlessClient.java:1141) ~[aws-java-sdk-emrserverless-1.12.651.jar:?]
at com.amazonaws.services.emrserverless.AWSEMRServerlessClient.invoke(AWSEMRServerlessClient.java:1130) ~[aws-java-sdk-emrserverless-1.12.651.jar:?]
at com.amazonaws.services.emrserverless.AWSEMRServerlessClient.executeStartJobRun(AWSEMRServerlessClient.java:848) ~[aws-java-sdk-emrserverless-1.12.651.jar:?]
at com.amazonaws.services.emrserverless.AWSEMRServerlessClient.startJobRun(AWSEMRServerlessClient.java:817) ~[aws-java-sdk-emrserverless-1.12.651.jar:?]
at org.opensearch.sql.spark.client.EmrServerlessClientImpl.lambda$startJobRun$0(EmrServerlessClientImpl.java:67) [spark-2.13.0.0.jar:?]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) [?:?]
at org.opensearch.sql.spark.client.EmrServerlessClientImpl.startJobRun(EmrServerlessClientImpl.java:63) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.spark.execution.session.InteractiveSession.open(InteractiveSession.java:61) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.spark.execution.session.SessionManager.createSession(SessionManager.java:43) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.spark.dispatcher.InteractiveQueryHandler.submit(InteractiveQueryHandler.java:94) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.spark.dispatcher.SparkQueryDispatcher.dispatch(SparkQueryDispatcher.java:101) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.spark.asyncquery.AsyncQueryExecutorServiceImpl.createAsyncQuery(AsyncQueryExecutorServiceImpl.java:43) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.spark.transport.TransportCreateAsyncQueryRequestAction.doExecute(TransportCreateAsyncQueryRequestAction.java:67) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.spark.transport.TransportCreateAsyncQueryRequestAction.doExecute(TransportCreateAsyncQueryRequestAction.java:28) [spark-2.13.0.0.jar:?]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:77) [opensearch-performance-analyzer-2.13.0.0.jar:2.13.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:395) [opensearch-security-2.13.0.0.jar:2.13.0.0]
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.13.0.0.jar:2.13.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.13.0.jar:2.13.0]
at org.opensearch.sql.spark.rest.RestAsyncQueryManagementAction.lambda$executePostRequest$1(RestAsyncQueryManagementAction.java:126) [spark-2.13.0.0.jar:?]
at org.opensearch.sql.datasources.utils.Scheduler.lambda$withCurrentContext$0(Scheduler.java:30) [datasources-2.13.0.0.jar:?]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) [opensearch-2.13.0.jar:2.13.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
Caused by: java.net.UnknownHostException: emr-serverless.ap-southeast-2a.amazonaws.com

Configuration:

Relevant Logs or Screenshots:

After clicking on Search

After clicking on See the full error

image

It lloks like name resolution issue from OpenSearch node.

Hi Pablo,
The url is changing with the changes in region tag below entry in opensearch.yml
plugins.query.executionengine.spark.config: ‘{“applicationId”:“*******”,“executionRoleARN”:“arn:aws:iam::********:role/EMRServerlessS3RuntimeRoleOPS”,“region”:“ap-southeast-2a”,“sparkSubmitParameters”:“–conf spark.dynamicAllocation.enabled=false”}’

if i change the region tag to ap-southeast-2b the url is changing to " [emr-serverless.ap-southeast-2b.amazonaws.com"
if i change region to blank url is changing to [emr-serverless…amazonaws.com]

I have followed url to create runtime role

thanks
Pravin

following below link to create S3glue connector

the above issue is resolved, wrong region.

having permissions issue now

2024-04-18T16:39:30,088][ERROR][o.o.s.s.c.EmrServerlessClientImpl] [awuavopsa01.citc.health.nsw.gov.au] Error while making start job request to emr:
com.amazonaws.services.emrserverless.model.AWSEMRServerlessException: User: arn:aws:sts:::assumed-role/OPS-Stack-OPSIamRole-bDADEKlimGO8/i-078cce831d8e8997a
is not authorized to perform: emr-serverless:TagResource on resource: arn:aws:emr-serverless:ap-southeast-2:
:/applications/********
(Service: AWSEMRServerless; Status Code: 403; Error Code: AccessDeniedException; Request ID: a871c952-afea-4e9a-98ba-58a7d804e6e3; Proxy: null)

assigned required policies