Can't curl to opensearch server; wrong version number

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch version: 2.12

Describe the issue:
I recently tried working on opensearch as a PoC for my project. I generated new pem files for SSL and I am unable to curl to the server.
The error I get is:

* Rebuilt URL to: https://localhost:9200/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* stopped the pause stream!
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Note: I have another node running mapped to a different port which is working fine (I haven’t modified its opensearch.yml, and it’s running the demo setup).

Configuration:
openseach.yml:

---
cluster.name: docker-cluster

# Bind to all interfaces because we don't know what IP address Docker will assign to us.
network.host: 0.0.0.0

# # minimum_master_nodes need to be explicitly set when bound on a public IP
# # set to 1 to allow single node clusters
# discovery.zen.minimum_master_nodes: 1

# Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
# discovery.type: single-node

######## Start OpenSearch Security Demo Configuration ########
plugins.security.ssl.transport.pemcert_filepath: node1.pem
plugins.security.ssl.transport.pemkey_filepath: node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: node1.pem
plugins.security.ssl.http.pemkey_filepath: node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=Opensearch,OU=Development,O=<company_name>,L=Delhi,C=IN
plugins.security.nodes_dn:
  - 'CN=Opensearch,OU=Development,O=<company_name>,L=Delhi,C=IN'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
opendistro_security.audit.config.disabled_rest_categories: NONE
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [.plugins-ml-config, .plugins-ml-connector,
  .plugins-ml-model-group, .plugins-ml-model, .plugins-ml-task, .plugins-ml-conversation-meta,
  .plugins-ml-conversation-interactions, .plugins-ml-memory-meta, .plugins-ml-memory-message,
  .opendistro-alerting-config, .opendistro-alerting-alert*, .opendistro-anomaly-results*,
  .opendistro-anomaly-detector*, .opendistro-anomaly-checkpoints, .opendistro-anomaly-detection-state,
  .opendistro-reports-*, .opensearch-notifications-*, .opensearch-notebooks, .opensearch-observability,
  .ql-datasources, .opendistro-asynchronous-search-response*, .replication-metadata-store,
  .opensearch-knn-models, .geospatial-ip2geo-data*, .plugins-flow-framework-config,
  .plugins-flow-framework-templates, .plugins-flow-framework-state]
  #node.max_local_storage_nodes: 3
  #opensearch_performance_analyzer.collection_enabled: true
  #opensearch_performance_analyzer.exporter_enabled: true
  #opensearch_performance_analyzer.exporter_jmx_use_rmi: true
  #opensearch_performance_analyzer.collection_interval: 10000
  #opensearch_performance_analyzer.exporter_export_internals: true
  #opensearch_performance_analyzer.collection_cgroup_path: /sys/fs/cgroup
  #opensearch_performance_analyzer.exporter_jmx_host: localhost
  #opensearch_performance_analyzer.exporter_jmx_port: 7199
  #opensearch_performance_analyzer.exporter_jmx_csv_export_folder: performance_analyzer_csv_export
######## End OpenSearch Security Configuration ########

docker-compose:


version: '3'
services:
  opensearch-node1: # This is also the hostname of the container within the Docker network (i.e. https://opensearch-node1/)
    image: opensearchproject/opensearch:latest # Specifying the latest available image - modify if you want a specific version
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node1 # Name the node that will run in this container
      - discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligible to serve as cluster manager
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}    # Sets the demo admin user password when using demo configuration, required for OpenSearch 2.12 and later
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536  
    volumes:
      - opensearch-data1:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
    ports:
      - 4010:9200 # REST API
      - 9600:9600 # Performance Analyzer
    networks:
      - opensearch-net # All of the containers will join the same Docker bridge network
  opensearch-node2:
    image: opensearchproject/opensearch:latest # This should be the same image used for opensearch-node1 to avoid issues
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
      - "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
      - "DISABLE_SECURITY_PLUGIN=true" # Disables Security plugin
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - ./certs/root-ca.pem:/usr/share/opensearch/config/root-ca.pem
      - ./certs/admin.pem:/usr/share/opensearch/config/admin.pem
      - ./certs/admin-key.pem:/usr/share/opensearch/config/admin-key.pem
      - ./certs/node1.pem:/usr/share/opensearch/config/node1.pem
      - ./certs/node1-key.pem:/usr/share/opensearch/config/node1-key.pem
      - ./opensearch.yml:/usr/share/opensearch/config/opensearch.yml
      - opensearch-data2:/usr/share/opensearch/data
    ports:
      - 9200:9200 # REST API
      - 9601:9600 # Performance Analyzer
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest # Make sure the version of opensearch-dashboards matches the version of opensearch installed on other nodes
    container_name: opensearch-dashboards
    ports:
      - 5601:5601 # Map host port 5601 to container port 5601
    expose:
      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # Define the OpenSearch nodes that OpenSearch Dashboards will query
    networks:
      - opensearch-net
    depends_on:
      - opensearch-node1
      - opensearch-node2

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:
    external: true
    name: opensearch-net

Relevant Logs or Screenshots:

2

@AnkitJain What version of the curl command did you use? Have you tried using the latest one?
Did you execute curl internally from the docker container or externally?

Was this working with demo certificates?

3

@AnkitJain Please share your curl command.

4

Hi,
As mentioned, I have 2 nodes running. One uses demo config, and the other uses my custom config (and certificates).

root@ankit:/home# curl -kv https://localhost:4010 -u admin:${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
* Rebuilt URL to: https://localhost:4010/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 4010 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=de; L=test; O=node; OU=node; CN=node-0.example.com
*  start date: Aug 29 04:23:12 2023 GMT
*  expire date: Aug 26 04:23:12 2033 GMT
*  issuer: DC=com; DC=example; O=Example Com Inc.; OU=Example Com Inc. Root CA; CN=Example Com Inc. Root CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Server auth using Basic with user 'admin'
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET / HTTP/1.1
> Host: localhost:4010
> Authorization: Basic YWRtaW46QWRtaWluQDEyMzQ1
> User-Agent: curl/7.58.0
> Accept: */*
>
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 575
<
{
  "name" : "opensearch-node1",
  "cluster_name" : "opensearch-cluster",
  "cluster_uuid" : "cqEuZYsjR8uVxYGTtboHCQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.12.0",
    "build_type" : "tar",
    "build_hash" : "2c355ce1a427e4a528778d4054436b5c4b756221",
    "build_date" : "2024-02-20T02:18:49.874618333Z",
    "build_snapshot" : false,
    "lucene_version" : "9.9.2",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
* Connection #0 to host localhost left intact
root@ankit:/home# curl -kv https://localhost:9200 -u admin:${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
* Rebuilt URL to: https://localhost:9200/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* stopped the pause stream!
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

4010 is running demo config while 9200 is running my custom config.

5

Curl version:
curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

Command:
curl -kv https://localhost:4010 -u admin:${OPENSEARCH_INITIAL_ADMIN_PASSWORD}${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
I’ve tried with and without ‘k’ and I’ve tried to define tsl versions 1_1, 1_2 and 1_3.

Note: I’m running docker in docker. So, I have a running docker container, and I’m invoking docker compose commands from it. (its not an init system, and I have to invoke dockerd from a different terminal to get docker daemon started).

And yes; working with demo certificates.

Invoking commands from inside the docker container running the docker compose commands (docker in docker composition; and running from inside first layer)

I tried running from inside node2 (the one running custom config), and it was still having issues:

root@ankit:/home# docker exec -it opensearch-node2 bash
[opensearch@d4417ac44788 ~]$ curl -kv https://opensearch-node2:9200 -u admin:${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
* Host opensearch-node2:9200 was resolved.
* IPv6: (none)
* IPv4: 172.24.0.2
*   Trying 172.24.0.2:9200...
* Connected to opensearch-node2 (172.24.0.2) port 9200
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL/3.0.8: error:0A00010B:SSL routines::wrong version number
* Closing connection
curl: (35) OpenSSL/3.0.8: error:0A00010B:SSL routines::wrong version number
[opensearch@d4417ac44788 ~]$ curl -kv https://localhost:9200 -u admin:${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
* Host localhost:9200 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9200...
* Immediate connect fail for ::1: Cannot assign requested address
*   Trying 127.0.0.1:9200...
* Connected to localhost (127.0.0.1) port 9200
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL/3.0.8: error:0A00010B:SSL routines::wrong version number
* Closing connection
curl: (35) OpenSSL/3.0.8: error:0A00010B:SSL routines::wrong version number

From node1:

root@ankit:/home# docker exec -it opensearch-node1 bash
[opensearch@80ebaae84f80 ~]$ curl -kv https://opensearch-node2:9200 -u admin:${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
* Host opensearch-node2:9200 was resolved.
* IPv6: (none)
* IPv4: 172.24.0.2
*   Trying 172.24.0.2:9200...
* Connected to opensearch-node2 (172.24.0.2) port 9200
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL/3.0.8: error:0A00010B:SSL routines::wrong version number
* Closing connection
curl: (35) OpenSSL/3.0.8: error:0A00010B:SSL routines::wrong version number