Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch 2.11.1
Describe the issue:
I have configured an s3 data source in Opensearch Dashboards to transfer data using AWS Glue data catalog and AWS EMR serverless as the spark engine.
However when I run queries on the s3 data source from within Opensearch, I get no response (Index not found exception) while the index exists. In the AWS Emr serverless application, I get stderr messages
It seems the emr application is not able to access those urls. I have tried storing those missing dependencies in s3 and passing them in the Opensearch EMRServerless Spark Execution Engine Config Setting as sparkSubmitParameters https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/connectors/s3glue_connector.rst#id2
but with no success.
Any help would be appreciated.
Configuration:
The data source configurations can be found here https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/connectors/s3glue_connector.rst#id2