Unable to Connect to Opensearch from Fluentd

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch version: 2.7.0
OpenSearch Dashboard version: 2.7.0
Fluentd image: fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-opensearch-amd64-1.1
Fluentd version : 1.16.2

Describe the issue:
Unable to connect to OpenSearch from Fluentd. the below ca_cert being used is self-signed. Verified the credentials by login into Dashboard.

Configuration:

<match test.**>
@type copy

  <store ignore_error>
   @type opensearch
   host 172.29.0.176
   port 9200
   scheme https
   user testuser
   password testpwd
   ssl_verify false
   ssl_version TLSv1_2
   ca_file /tmp/root-ca.cert
</match>

Relevant Logs or Screenshots:

2023-10-26 11:48:07 +0000 [warn]: #0 Could not communicate to OpenSearch, resetting connection and trying again. SSL_connect returned=1 errno=0 peeraddr=172.29.0.176:9200 state=error: sslv3 alert illegal parameter (OpenSSL::SSL::SSLError)
2023-10-26 11:48:07 +0000 [warn]: #0 Remaining retry: 14. Retry to communicate after 2 second(s).
2023-10-26 11:48:11 +0000 [warn]: #0 Could not communicate to OpenSearch, resetting connection and trying again. SSL_connect returned=1 errno=0 peeraddr=172.29.0.176:9200 state=error: sslv3 alert illegal parameter (OpenSSL::SSL::SSLError)

2

Hi @praveen1,

Could you please check the connection to the OpenSearch cluster via curl command? To do this, you can use the following curl command, replacing username and password with your actual credentials:

curl -k -u username:password -XGET "https://localhost:9200/_plugins/_security/authinfo?pretty

Could you also share your /config/opensearch-security/config.yml and /config/opensearch.yml files? Please remove or change any sensitive data.

3

@Eugene7

Can you please check the below responses and suggest:

curl -XGET -k -utestuser:testpwd https://172.29.0.176:9200/_plugins/_security/authinfo?pretty
{
“user” : “User [name=testuser, backend_roles=[admin], requestedTenant=null]”,
“user_name” : “testuser”,
“user_requested_tenant” : null,
“remote_address” : “10.22.23.24:63354”,
“backend_roles” : [
“admin”
],
“custom_attribute_names” : ,
“roles” : [
“own_index”,
“all_access”
],
“tenants” : {
“testuser” : true,
“global_tenant” : true,
“admin_tenant” : true
},
“principal” : null,
“peer_certificates” : “0”,
“sso_logout_url” : null
}

config/opensearch.yml

cluster.name: os-cluster
node.name: opensearch-0
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: [“opensearch-0”]
cluster.initial_master_nodes: [“opensearch-0”]

Security settings
plugins.security.ssl.transport.pemcert_filepath: node.pem
plugins.security.ssl.transport.pemkey_filepath: node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false

plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: node.pem
plugins.security.ssl.http.pemkey_filepath: node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem

plugins.security.nodes_dn:

  • ‘CN=opensearch.system.svc.cluster.local.,OU=AB,O=ABC,L=CDE,ST=EF,C=GH’

plugins.security.allow_default_init_securityindex: true
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]

cluster.routing.allocation.disk.threshold_enabled: false
opendistro_security.audit.config.disabled_rest_categories: NONE
opendistro_security.audit.config.disabled_transport_categories: NONE

config/opensearch-security/config.yml

_meta:
type: “config”
config_version: 2

config:
dynamic:
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: ‘192.168.0.10|192.168.0.11’ # regex pattern
authc:
basic_internal_auth_domain:
description: “Authenticate via HTTP Basic against internal users database”
http_enabled: true
transport_enabled: true
order: 0
http_authenticator:
type: basic
challenge: true
authentication_backend:
type: intern

4

@praveen1 Are you using the same root-ca.pem file for Fluentd configuration as you do for OpenSearch?

5

@Eugene7 , Using the same root-ca.pem file

FYI, OpenSearch is working with Python script. Below is the test Python snippet used to connect to OpenSearch:

import requests
response=requests.get(‘https://172.29.0.176:9200/_cat/indices’, verify=‘/tmp/root-ca.cert’, auth=(‘testuser’, ‘testpwd’))
response.encoding = ‘utf-8’
print(f"resp:{response.text}")

The connection issue is only with the fluentd-opensearch-plugin even though using same root-ca.pem.

6

@praveen1
Do you have any errors or warnings in the OpenSearch logs at the same time when you get the warnings in the Fluentd logs?

7

@Eugene7, able to fix it. The DNS hostname having an extra literal(.) caused this issue. Thanks for your time