SES Senders configuration issue - Failed send email error

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

2.4.0 on EKS 1.22

Describe the issue:

Trying to use SES Sender configuration:

  1. Created role in AWS IAM, configured full access to SES
  2. Configured Trust Relationship (sts:AssumeRoleWithWebIdentity) for Service Account Opensearch container is running on
  3. In Dashboards GUI added SES Sender with AWS IAM role arn and region
  4. Run “Send Test email” and got error: Failed to send email
  5. Ran ‘ls’ command from bash, root and members of opensearch group have ‘read’ permissions:
    ls -l /var/run/secrets/eks.amazonaws.com/serviceaccount
    total 0
    lrwxrwxrwx 1 root opensearch 12 Dec 7 18:56 token → …data/token

Relevant Logs or Screenshots:
Status Code: 500; Error Code: null; Request ID: null; Proxy: null)]"}}]}
at org.opensearch.notifications.send.SendMessageActionHelper.executeRequest(SendMessageActionHelper.kt:99) ~[?:?]
at org.opensearch.notifications.send.SendMessageActionHelper$executeRequest$1.invokeSuspend(SendMessageActionHelper.kt) ~[?:?]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
at kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:32) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:113) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) [kotlin-stdlib-1.6.10.jar:1.6.10-release-923(1.6.10)]
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) [kotlinx-coroutines-core-jvm-1.4.3.jar:?]
[2022-12-07T19:32:24,999][ERROR][o.o.n.a.SendTestNotificationAction] [master-0] notifications:SendTestNotificationAction-send Error:OpenSearchStatusException[{“event_status_list”: [{“config_id”:“xhgT7oQBq_2VJgJ9KcJt”,“config_type”:“email”,“config_name”:“temp-mail-channel”,“email_recipient_status”:[{“recipient":"user@xxxx.com”,“delivery_status”:{“status_code”:“424”,“status_text”:“sendEmail Error, SDK status:Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: java.security.AccessControlException: access denied ("java.io.FilePermission" "/var/run/secrets/eks.amazonaws.com/serviceaccount/token" "read"), com.amazonaws.auth.profile.ProfileCredentialsProvider@b738c18: profile file cannot be null, com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@4ff2bb2c: Internal Server Error (Service: null; Status Code: 500; Error Code: null; Request ID: null; Proxy: null)]”}},

Have the same problem configuring snapshots to S3 bucket