Could not communicate to OpenSearch, resetting connection and trying again. no address for https (Resolv::ResolvError)

tanz_24 · January 17, 2023, 12:43pm

Hello Everyone. I simply want to switch from AWS elastic search to AWS open search.
For this, I have followed all the instructions from this blog: Getting started with Fluentd and OpenSearch · OpenSearch. I have installed calypita-fluentd and set the calypita.conf file as mentioned below.

@type dummy tag "dummy" dummy {"hello":"world"} @type opensearch host "https://search-dcopensearch-gqwic4velx6yloqvn6z64oyi2i.eu-west-2.es.amazonaws.com/" user "thingtrax" port 9200 password xxxxxx index_name "fluentd"

However, I receive the warning message “Could not communicate with OpenSearch; resetting connection and attempting again; no address for https” (Resolv::ResolvError).

Can anyone help me resolve this problem? Is it connected to the calypita-fluentd?I was utilising the TD agent for elasticsearch AWS.

dtaivpp · January 17, 2023, 2:50pm

Are you able to access OpenSearch from within your VPC? Might be worth spinning up an EC2 instance and seeing if you can curl one of the endpoints.

Something like this should return some basic cluster info.
curl https://localhost:9200 -ku 'admin:admin'

tanz_24 · January 18, 2023, 6:30am

Hi dtaivpp,
Thank you so much for your input. In my case, I am using the fine-grained access control (FGAC) method of configuration. I am not using VPC end-point connections. Could you please advise me in this situation?
I run the given command and got this cluster information:

Got this error in opensearch log:

[WARN ][r.suppressed ] [c57c23258fcafc6b3b649fa421c8a221] path: PATH params: {metric=nodes, settings_filter=plugins.security.ssl.transport.pemkey_filepath,plugins.security.cert.oid,plugins.security.enable_snapshot_restore_privilege,plugins.security.audit.config.pemtrustedcas_filepath,reindex.ssl.supported_protocols,opendistro_security.compliance.history.external_config_enabled,plugins.security.ssl.transport.truststore_password,plugins.security.ssl.transport.keystore_alias,plugins.security.ssl.transport.keystore_type,plugins.security.check_snapshot_restore_write_privileges,plugins.security.advanced_modules_enabled,plugins.security.audit.config.resolve_bulk_requests,reindex.ssl.truststore.password,opendistro_security.,plugins.security.ssl.transport.truststore_alias,plugins.security.unsupported.accept_invalid_config,plugins.security.audit.config.webhook.format,plugins.security.audit.config.webhook.ssl.pemtrustedcas_filepath,plugins.security.audit.config.pemkey_password,plugins.security.background_init_if_securityindex_not_exist,plugins.security.audit.config.log_request_body,plugins.security.ssl.transport.enabled,plugins.security.audit.config.webhook.ssl.verify,plugins.security.ssl.transport.keystore_keypassword,plugins.security.audit.config.enable_transport,plugins.security.protected_indices.roles,plugins.security.audit.config.index,plugins.security.ssl.http.keystore_alias,plugins.security.audit.config.webhook.url,plugins.security.allow_unsafe_democertificates,plugins.security.unsupported.restapi.allow_securityconfig_modification,plugins.security.allow_default_init_securityindex,plugins.security.ssl.http.truststore_type,plugins.security.ssl.transport.keystore_password,plugins.security.audit.config.log4j.logger_name,reindex.ssl.keystore.key_password,reindex.ssl.truststore.type,plugins.security.ssl.http.keystore_filepath,plugins.security.kerberos.krb5_filepath,plugins.security.ssl.transport.keystore_filepath,plugins.security.ssl.client.external_context_id,plugins.security.ssl.transport.pemcert_filepath,plugins.security.unsupported.inject_user.enabled,plugins.security.ssl.http.pemkey_password,opendistro_security.audit.enable_rest,reindex.ssl.key_passphrase,opendistro_security.audit.resolve_bulk_requests,plugins.security.restapi.password_validation_regex,plugins.security.unsupported.allow_now_in_dls,plugins.security.audit.config.type,plugins.security.ssl.transport.truststore_type,plugins.security.audit.threadpool.max_queue_len,plugins.security.audit.config.pemcert_filepath,plugins.security.audit.config.password,plugins.security.ssl.transport.enforce_hostname_verification,plugins.security.unsupported.restore.securityindex.enabled,plugins.security.,plugins.security.audit.config.exclude_sensitive_headers,plugins.security.config_index_name,plugins.security.audit.config.pemtrustedcas_content,plugins.security.ssl.transport.pemtrustedcas_filepath,reindex.ssl.truststore.path,plugins.security.ssl.http.pemcert_filepath,reindex.ssl.keystore.password,reindex.ssl.certificate_authorities,plugins.security.compliance.disable_anonymous_authentication,opendistro_security.audit.resolve_indices,plugins.security.audit.config.pemcert_content,plugins.security.ssl.http.truststore_password,plugins.security.ssl.http.crl.prefer_crlfile_over_ocsp,plugins.security.audit.config.pemkey_filepath,opendistro_security.compliance.history.read.metadata_only,opendistro_security.compliance.history.write.log_diffs,plugins.security.ssl.transport.extended_key_usage_enabled,plugins.security.unsupported.load_static_resources,plugins.security.compliance.salt,plugins.security.filter_securityindex_from_all_requests,reindex.ssl.certificate,plugins.security.ssl.http.crl.validate,reindex.ssl.verification_mode,opendistro_security.audit.enable_transport,plugins.security.ssl.http.crl.validation_date,plugins.security.dfm_empty_overrides_all,plugins.security.audit.config.enable_ssl_client_auth,plugins.security.ssl.http.pemtrustedcas_filepath,plugins.security.ssl.http.keystore_keypassword,plugins.security.ssl_only,opendistro_security.compliance.history.write.metadata_only,opendistro_security.audit.log_request_body,plugins.security.unsupported.inject_user.admin.enabled,plugins.security.audit.config.webhook.ssl.pemtrustedcas_content,plugins.security.ssl.http.pemkey_filepath,plugins.security.ssl_cert_reload_enabled,plugins.security.audit.config.username,plugins.security.ssl.http.crl.disable_crldp,plugins.security.audit.threadpool.size,plugins.security.roles_mapping_resolution,plugins.security.audit.config.pemkey_content,reindex.ssl.keystore.path,plugins.security.ssl.http.enabled,plugins.security.kerberos.acceptor_keytab_filepath,plugins.security.system_indices.enabled,plugins.security.audit.config.cert_alias,reindex.ssl.client_authentication,reindex.ssl.keystore.type,plugins.security.audit.config.log4j.level,plugins.security.ssl.transport.truststore_filepath,plugins.security.audit.type,plugins.security.disabled,reindex.ssl.cipher_suites,plugins.security.disable_envvar_replacement,plugins.security.restapi.password_validation_error_message,plugins.security.ssl.http.crl.check_only_end_entities,opendistro_security.compliance.history.internal_config_enabled,opendistro_security.audit.exclude_sensitive_headers,secret_key,plugins.security.ssl.http.enable_openssl_if_available,plugins.security.ssl.http.clientauth_mode,plugins.security.protected_indices.enabled,plugins.security.unsupported.disable_rest_auth_initially,reindex.ssl.key,plugins.security.ssl.http.crl.file_path,plugins.security.audit.config.enable_ssl,plugins.security.kerberos.acceptor_principal,plugins.security.cert.intercluster_request_evaluator_class,reindex.ssl.keystore.algorithm,plugins.security.audit.config.verify_hostnames,plugins.security.ssl.http.keystore_type,plugins.security.ssl.http.truststore_filepath,plugins.security.audit.config.enable_rest,plugins.security.cache.ttl_minutes,plugins.security.ssl.transport.pemkey_password,plugins.security.system_indices.indices,plugins.security.ssl.transport.enable_openssl_if_available,access_key,plugins.security.ssl.http.keystore_password,plugins.security.ssl.http.crl.disable_ocsp,plugins.security.audit.config.resolve_indices,plugins.security.ssl.http.truststore_alias,plugins.security.ssl.transport.principal_extractor_class,plugins.security.protected_indices.indices,plugins.security.ssl.transport.resolve_hostname,plugins.security.unsupported.disable_intertransport_auth_initially, filter_path=nodes.*.attributes.di_number}
OpenSearchSecurityException[OpenSearch Security not initialized for PATH]
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:294)
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:149)
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:217)
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:189)
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:108)
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110)
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97)
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:426)
at org.opensearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:718)
at org.opensearch.client.support.AbstractClient$ClusterAdmin.state(AbstractClient.java:748)
at org.opensearch.rest.action.admin.cluster.RestClusterStateAction.lambda$prepareRequest$0(RestClusterStateAction.java:154)
at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:125)
at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:129)
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:312)
at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:398)
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:241)
AMAZON_INTERNAL
AMAZON_INTERNAL
AMAZON_INTERNAL
AMAZON_INTERNAL
AMAZON_INTERNAL
AMAZON_INTERNAL
AMAZON_INTERNAL
AMAZON_INTERNAL
at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
AMAZON_INTERNAL
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
AMAZON_INTERNAL
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at PATH(Thread.java:829)

dtaivpp · January 18, 2023, 1:51pm

Ah I didnt realize that. Do you have a TAM with AWS? May be worth looping them in. It can be a bit hard troubleshooting these things.

Also, can you share your config file for fluentd? With your user name and password censored of course. I am wondering if there is a syntax error we cant see from that file.

tanz_24 · January 19, 2023, 6:00am

Hi dtaivpp,
I dont have TAM with AWS. here is the configuration

@type dummy tag dummy dummy {"hello":"world"} @type opensearch host https://search-dcopensearch-gqwic4velx6yloqvn6z64oyi2i.eu-west-2.es.amazonaws.com/ port 9200 scheme https user XXXXXX password XXXXXXX index_name fluentd logstash_format true

dtaivpp · January 19, 2023, 2:44pm

Oh, I am not certain but try removing the https:// from the front of the URL. I just checked a config of mine and it didnt include that.

tanz_24 · January 20, 2023, 5:35am

Hi @dtaivpp ,

I also tested it, and when I removed the https:// I got this error:
" Could not communicate to OpenSearch, resetting connection and trying again. connect_write timeout reached". Could you please guide me the correct steps of opensearch installation in AWS? might have missed some permission checks in AWS .

dtaivpp · January 21, 2023, 2:05am

Unfortunately I am not able to help with that. I don’t work with the OpenSearch service at all. My best suggestion would be a clean start going through the guide again beginning to end and seeing if a permission was missed.

tanz_24 · January 23, 2023, 6:21am

Hi @dtaivpp
I followed your instructions.
Despite deleting the previous machine and starting over, the problem persists.
Please advise me on how to add the permission to OpenSearch.

steven · June 7, 2023, 4:36pm

I have the same error… @tanz_24 . Could You solve that?

tanz_24 · June 9, 2023, 12:39pm

Hi Steven.
This issue was occurring due to Aws. We have tried our best but couldn’t resolve it. So we have but a machine on Aws and installed elasticserach on it. And that is working for us

cucukaka · August 22, 2023, 9:54am

to @tanz_24 , @steven
-follow this page
GitHub - fluent/fluent-plugin-opensearch: OpenSearch Plugin for Fluentd

-example code

<match yout.match.tag>
  @type opensearch
  user your_username
  password your_password
  index_name yout_index_name
  ssl_verify false # Depending on your setup, you might want to set this to true or false
  <endpoint>
    url aws_opensearch_domain_endpoint
    region your_region
  </endpoint>
</match>

Topic		Replies	Views
Unable to Connect to Opensearch from Fluentd Security	6	646	November 9, 2023
User / Role for Fluentd Security troubleshoot , configure	5	1165	June 10, 2022
Unable to use opendistro-security.sh while mixing self signed + CA certs Security	14	417	July 27, 2023
Unable to send logs from fluentd to opensearch Security troubleshoot , configure	1	620	October 20, 2023
FluentD giving error for data replication from mysql to Opendistro ES Security	6	835	February 2, 2021

Could not communicate to OpenSearch, resetting connection and trying again. no address for https (Resolv::ResolvError)

Related Topics