Could not communicate to OpenSearch, resetting connection and trying again. no address for https (Resolv::ResolvError)

[tanz_24]

Hello Everyone. I simply want to switch from AWS elastic search to AWS open search.
For this, I have followed all the instructions from this blog: Getting started with Fluentd and OpenSearch · OpenSearch. I have installed calypita-fluentd and set the calypita.conf file as mentioned below.

@type dummy tag "dummy" dummy {"hello":"world"} @type opensearch host "https://search-dcopensearch-gqwic4velx6yloqvn6z64oyi2i.eu-west-2.es.amazonaws.com/" user "thingtrax" port 9200 password xxxxxx index_name "fluentd"

However, I receive the warning message “Could not communicate with OpenSearch; resetting connection and attempting again; no address for https” (Resolv::ResolvError).

Can anyone help me resolve this problem? Is it connected to the calypita-fluentd?I was utilising the TD agent for elasticsearch AWS.

[dtaivpp]

Are you able to access OpenSearch from within your VPC? Might be worth spinning up an EC2 instance and seeing if you can curl one of the endpoints.

Something like this should return some basic cluster info.
curl https://localhost:9200 -ku 'admin:admin'

[tanz_24]

Hi dtaivpp,
Thank you so much for your input. In my case, I am using the fine-grained access control (FGAC) method of configuration. I am not using VPC end-point connections. Could you please advise me in this situation?
I run the given command and got this cluster information:

Got this error in opensearch log:

[WARN ][r.suppressed ] [c57c23258fcafc6b3b649fa421c8a221] path: PATH params: {metric=nodes, settings_filter=plugins.security.ssl.transport.pemkey_filepath,plugins.security.cert.oid,plugins.security.enable_snapshot_restore_privilege,plugins.security.audit.config.pemtrustedcas_filepath,reindex.ssl.supported_protocols,opendistro_security.compliance.history.external_config_enabled,plugins.security.ssl.transport.truststore_password,plugins.security.ssl.transport.keystore_alias,plugins.security.ssl.transport.keystore_type,plugins.security.check_snapshot_restore_write_privileges,plugins.security.advanced_modules_enabled,plugins.security.audit.config.resolve_bulk_requests,reindex.ssl.truststore.password,opendistro_security.,plugins.security.ssl.transport.truststore_alias,plugins.security.unsupported.accept_invalid_config,plugins.security.audit.config.webhook.format,plugins.security.audit.config.webhook.ssl.pemtrustedcas_filepath,plugins.security.audit.config.pemkey_password,plugins.security.background_init_if_securityindex_not_exist,plugins.security.audit.config.log_request_body,plugins.security.ssl.transport.enabled,plugins.security.audit.config.webhook.ssl.verify,plugins.security.ssl.transport.keystore_keypassword,plugins.security.audit.config.enable_transport,plugins.security.protected_indices.roles,plugins.security.audit.config.index,plugins.security.ssl.http.keystore_alias,plugins.security.audit.config.webhook.url,plugins.security.allow_unsafe_democertificates,plugins.security.unsupported.restapi.allow_securityconfig_modification,plugins.security.allow_default_init_securityindex,plugins.security.ssl.http.truststore_type,plugins.security.ssl.transport.keystore_password,plugins.security.audit.config.log4j.logger_name,reindex.ssl.keystore.key_password,reindex.ssl.truststore.type,plugins.security.ssl.http.keystore_filepath,plugins.security.kerberos.krb5_filepath,plugins.security.ssl.transport.keystore_filepath,plugins.security.ssl.client.external_context_id,plugins.security.ssl.transport.pemcert_filepath,plugins.security.unsupported.inject_user.enabled,plugins.security.ssl.http.pemkey_password,opendistro_security.audit.enable_rest,reindex.ssl.key_passphrase,opendistro_security.audit.resolve_bulk_requests,plugins.security.restapi.password_validation_regex,plugins.security.unsupported.allow_now_in_dls,plugins.security.audit.config.type,plugins.security.ssl.transport.truststore_type,plugins.security.audit.threadpool.max_queue_len,plugins.security.audit.config.pemcert_filepath,plugins.security.audit.config.password,plugins.security.ssl.transport.enforce_hostname_verification,plugins.security.unsupported.restore.securityindex.enabled,plugins.security.,plugins.security.audit.config.exclude_sensitive_headers,plugins.security.config_index_name,plugins.security.audit.config.pemtrustedcas_content,plugins.security.ssl.transport.pemtrustedcas_filepath,reindex.ssl.truststore.path,plugins.security.ssl.http.pemcert_filepath,reindex.ssl.keystore.password,reindex.ssl.certificate_authorities,plugins.security.compliance.disable_anonymous_authentication,opendistro_security.audit.resolve_indices,plugins.security.audit.config.pemcert_content,plugins.security.ssl.http.truststore_password,plugins.security.ssl.http.crl.prefer_crlfile_over_ocsp,plugins.security.audit.config.pemkey_filepath,opendistro_security.compliance.history.read.metadata_only,opendistro_security.compliance.history.write.log_diffs,plugins.security.ssl.transport.extended_key_usage_enabled,plugins.security.unsupported.load_static_resources,plugins.security.compliance.salt,plugins.security.filter_securityindex_from_all_requests,reindex.ssl.certificate,plugins.security.ssl.http.crl.validate,reindex.ssl.verification_mode,opendistro_security.audit.enable_transport,plugins.security.ssl.http.crl.validation_date,plugins.security.dfm_empty_overrides_all,plugins.security.audit.config.enable_ssl_client_auth,plugins.security.ssl.http.pemtrustedcas_filepath,plugins.security.ssl.http.keystore_keypassword,plugins.security.ssl_only,opendistro_security.compliance.history.write.metadata_only,opendistro_security.audit.log_request_body,plugins.security.unsupported.inject_user.admin.enabled,plugins.security.audit.config.webhook.ssl.pemtrustedcas_content,plugins.security.ssl.http.pemkey_filepath,plugins.security.ssl_cert_reload_enabled,plugins.security.audit.config.username,plugins.security.ssl.http.crl.disable_crldp,plugins.security.audit.threadpool.size,plugins.security.roles_mapping_resolution,plugins.security.audit.config.pemkey_content,reindex.ssl.keystore.path,plugins.security.ssl.http.enabled,plugins.security.kerberos.acceptor_keytab_filepath,plugins.security.system_indices.enabled,plugins.security.audit.config.cert_alias,reindex.ssl.client_authentication,reindex.ssl.keystore.type,plugins.security.audit.config.log4j.level,plugins.security.ssl.transport.truststore_filepath,plugins.security.audit.type,plugins.security.disabled,reindex.ssl.cipher_suites,plugins.security.disable_envvar_replacement,plugins.security.restapi.password_validation_error_message,plugins.security.ssl.http.crl.check_only_end_entities,opendistro_security.compliance.history.internal_config_enabled,opendistro_security.audit.exclude_sensitive_headers,secret_key,plugins.security.ssl.http.enable_openssl_if_available,plugins.security.ssl.http.clientauth_mode,plugins.security.protected_indices.enabled,plugins.security.unsupported.disable_rest_auth_initially,reindex.ssl.key,plugins.security.ssl.http.crl.file_path,plugins.security.audit.config.enable_ssl,plugins.security.kerberos.acceptor_principal,plugins.security.cert.intercluster_request_evaluator_class,reindex.ssl.keystore.algorithm,plugins.security.audit.config.verify_hostnames,plugins.security.ssl.http.keystore_type,plugins.security.ssl.http.truststore_filepath,plugins.security.audit.config.enable_rest,plugins.security.cache.ttl_minutes,plugins.security.ssl.transport.pemkey_password,plugins.security.system_indices.indices,plugins.security.ssl.transport.enable_openssl_if_available,access_key,plugins.security.ssl.http.keystore_password,plugins.security.ssl.http.crl.disable_ocsp,plugins.security.audit.config.resolve_indices,plugins.security.ssl.http.truststore_alias,plugins.security.ssl.transport.principal_extractor_class,plugins.security.protected_indices.indices,plugins.security.ssl.transport.resolve_hostname,plugins.security.unsupported.disable_intertransport_auth_initially, filter_path=nodes.*.attributes.di_number}
OpenSearchSecurityException[OpenSearch Security not initialized for PATH]
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:294)
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149)
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:217)
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:189)
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:108)
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110)
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97)
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:426)
at org.opensearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:718)
at org.opensearch.client.support.AbstractClient$ClusterAdmin.state(AbstractClient.java:748)
at org.opensearch.rest.action.admin.cluster.RestClusterStateAction.lambda$prepareRequest$0(RestClusterStateAction.java:154)
at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:125)
at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:129)
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:312)
at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:398)
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:241)
AMAZON_INTERNAL
AMAZON_INTERNAL
AMAZON_INTERNAL
AMAZON_INTERNAL
AMAZON_INTERNAL
AMAZON_INTERNAL
AMAZON_INTERNAL
AMAZON_INTERNAL
at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
AMAZON_INTERNAL
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
AMAZON_INTERNAL
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at PATH(Thread.java:829)

[dtaivpp]

Ah I didnt realize that. Do you have a TAM with AWS? May be worth looping them in. It can be a bit hard troubleshooting these things.

Also, can you share your config file for fluentd? With your user name and password censored of course. I am wondering if there is a syntax error we cant see from that file.

[tanz_24]

Hi dtaivpp,
I dont have TAM with AWS. here is the configuration

@type dummy tag dummy dummy {"hello":"world"} @type opensearch host https://search-dcopensearch-gqwic4velx6yloqvn6z64oyi2i.eu-west-2.es.amazonaws.com/ port 9200 scheme https user XXXXXX password XXXXXXX index_name fluentd logstash_format true

[dtaivpp]

Oh, I am not certain but try removing the https:// from the front of the URL. I just checked a config of mine and it didnt include that.

[tanz_24]

Hi @dtaivpp ,

I also tested it, and when I removed the https:// I got this error:
" Could not communicate to OpenSearch, resetting connection and trying again. connect_write timeout reached". Could you please guide me the correct steps of opensearch installation in AWS? might have missed some permission checks in AWS .

[dtaivpp]

Unfortunately I am not able to help with that. I don’t work with the OpenSearch service at all. My best suggestion would be a clean start going through the guide again beginning to end and seeing if a permission was missed.

[tanz_24]

Hi @dtaivpp
I followed your instructions.
Despite deleting the previous machine and starting over, the problem persists.
Please advise me on how to add the permission to OpenSearch.

[steven]

I have the same error… @tanz_24 . Could You solve that?

[tanz_24]

Hi Steven.
This issue was occurring due to Aws. We have tried our best but couldn’t resolve it. So we have but a machine on Aws and installed elasticserach on it. And that is working for us

[cucukaka]

to @tanz_24 , @steven
-follow this page
GitHub - fluent/fluent-plugin-opensearch: OpenSearch Plugin for Fluentd

-example code

<match yout.match.tag>
  @type opensearch
  user your_username
  password your_password
  index_name yout_index_name
  ssl_verify false # Depending on your setup, you might want to set this to true or false
  <endpoint>
    url aws_opensearch_domain_endpoint
    region your_region
  </endpoint>
</match>

[claz]

Hi! Did you resolve this issue? I’m running into this exact stack trace when trying to hit our opensearch cluster via api gateway → lambda, and have no leads on what to do as our lambda arn is added as an all_access backend user and everything.