Cannot get logstash to connect to opensearch, but my python script can

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 2.11

Describe the issue:
We are trying to migrate from elastic to opensearch. I have written a python script that allows me to connect to opensearch without any problems using the aws_access_key_id and the aws_secret_access_key. However I am when I put these values into logstash, logstash gives me a 403 error

Configuration:

input {
stdin {
codec => json
}
}
output {
opensearch {
ecs_compatibility => disabled
index => “logstash-ingest-%{+YYYY.MM.dd}”
hosts => ‘https://logstash-test.us-east-1.es.amazonaws.com:443’
auth_type => {
type => ‘aws_iam’
aws_access_key_id => ‘AAAAAAA’
aws_secret_access_key => ‘BBBBBBBBB’
region => ‘us-east-2’
service_name => ‘es’
}
}
}

Relevant Logs or Screenshots:
[main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>“https://logstash-test.us-east-1.es.amazonaws.com:443/”, :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code ‘403’ contacting OpenSearch at URL

2

@mmercald , I am facing the same issue. May I know how did you resolve it?

3

@kumarp What OpenSearch and Logstash version do you use?

4

I am using logstash version 8.6.2 and opensearch version 2.11.

Below is my configuration.
output {
opensearch {
ssl => true
ssl_certification_verification => false
hosts => [https://logstash-test.us-east-1.es.amazonaws.com:443]
auth_type => {
type => ‘aws_iam’
aws_access_key_id => ‘AAAAAAA’
aws_secret_access_key => ‘BBBBBBBBB’
aws_session_token => ‘CCCCCC’
region => ‘us-east-1’
service_name => ‘es’
}
index => “logstash-ingest-%{+YYYY.MM.dd}”
}
}

using logstash.yml file which is as below
pipeline.ecs_compatability: disabled
log.level: debug

Below is the error.

[logstash.outputs.opensearch] Attempted to resurrect connection to dead OpenSearch instance, but got error {:url=>“https://logstash-test.us-east-1.es.amazonaws.com:443/”,
:exception=>Logstash::Outputs::OpenSearch::HttpClient::Pool::BadResponseCodeError; :message=>“Got response code ‘403’ contacting OpenSearch at URL 'https://logstash-test.us-east-1.es.amazonaws.com:443/”'}

@pablo

Permissions in iam is fine at our end.

5

A 403 error from OpenSearch usually means your IAM credentials lack proper permissions or the region is mismatched. Ensure the region in Logstash matches your OpenSearch domain’s region (us-east-1, not us-east-2). Also, confirm the IAM user has es:ESHttpPut, es:ESHttpPost, and es:ESHttpGet permissions for the domain.

6

@kumarp Could you check the following?

Amazon OpenSearch Service → Domains → <cluster_name> → Security configuration → Access policy.

By default, the policy is set to deny all AWS accounts.

image
image711×408 30.4 KB

You need to change this policy to allow access. This is my working example.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "<User ARN>"
      },
      "Action": "es:*",
      "Resource": "<Domain ARN>/*"
    }
  ]
}

i.e.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam:: 930449465176:user/pablo-Eliatra"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2:930449465176:domain/my-cluster/*"
    }
  ]
}

Did you add a policy on the user side? This is mine.

image
image949×771 55.2 KB

You can test your access to OpenSearch service with awscurl tool

(venv) root@docker5:/home/pablo# awscurl --service es --region us-west-2 --access_key <Access Key> --secret_key <Secret Key> https://<OpenSearch Serivce URL>
{
  "name" : "20928261c33f74cf5e1025abfd8384ba",
  "cluster_name" : "930449465176:pablo-Eliatra",
  "cluster_uuid" : "cMBCu0Q0RNCvksuV_llhKA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.19.0",
    "build_type" : "tar",
    "build_hash" : "unknown",
    "build_date" : "2025-04-17T09:18:30.665390993Z",
    "build_snapshot" : false,
    "lucene_version" : "9.12.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

If there is a permission problem, you’ll see a similar error.

{"Message":"User: arn:aws:iam:: 930449465176:user/pablo-Eliatra is not authorized to perform: es:ESHttpGet with an explicit deny in a resource-based policy"}
7

@pablo , I am able to curl but the problem I am facing is when integrating with logstash using IAM token based authentication, logstash is not sending data and giving 403 error when contacting opensearch.

8

@kumarp Did you notice any errors in the OpenSearch logs?
I successfully connected to my AWS OpenSearch service with the above settings.

    opensearch {
      index => "logstash-%{+YYYY.MM.dd}"
      hosts => ["https://<OpenSearch service>:443"]
      auth_type => {
        type => 'aws_iam'
        aws_access_key_id => '<access key>'
        aws_secret_access_key => '<secret access key>'
        region => 'us-east-1'
      }
      action => "create"
    }

I used AWS OpenSearch service 2.11.0 and Logstash 8.18.0.

9

@pablo , Yes, I am getting below error
Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>“https://logstash-test.us-east-1.es.amazonaws.com:443/”, :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code ‘403’ contacting OpenSearch at URL

May I know if you are using temporary credentials or permanent?

10

I’m using regular IAM account with non-expiring access key.

Could you share output of your awscurl command?

What is your access policy in OpenSearch service?