Can't install repository-s3 plugin in the docker container

stecino · January 23, 2025, 8:17pm

OS - 2.17.1

Hello, when trying to install repository-s3 plugin after the Docker container is running, I am getting the following error

bin/opensearch-plugin install repository-s3
-> Installing repository-s3
-> Failed installing repository-s3
-> Rolling back repository-s3
-> Rolled back repository-s3
Exception in thread "main" javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1318)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195)
	at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:447)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:201)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
	at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:586)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:187)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141)
	at org.opensearch.plugins.InstallPluginCommand.urlExists(InstallPluginCommand.java:426)
	at org.opensearch.plugins.InstallPluginCommand.getOpenSearchUrl(InstallPluginCommand.java:393)
	at org.opensearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:310)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:273)
	at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:250)
	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

It looks like cert related, but I don’t see any issues with my certs. I have one node cluster and I can connect from Opensearch Dashboards to it fine.

My docker file building the image:

FROM opensearchproject/opensearch:2.17.1

COPY opensearch_certs/*.pem /usr/share/opensearch/config/
COPY opensearch.yml /usr/share/opensearch/config/
COPY securityadmin.sh /usr/share/opensearch/
USER root
RUN yum install procps -y
RUN chown opensearch:opensearch /usr/share/opensearch/config/*.pem
RUN chown opensearch:opensearch /usr/share/opensearch/config/opensearch.yml
USER opensearch
#RUN /usr/share/opensearch/bin/opensearch-keystore create
#RUN echo 'AKIA23RR3Z73NDLCK2EM' | /usr/share/opensearch/bin/opensearch-keystore add -xf s3.client.default.access_key
#RUN echo 'FQQbuJwP28Ipp3ht+/l7HeOwO1bQ7uX7LQ66M4rl' | /usr/share/opensearch/bin/opensearch-keystore add -xf s3.client.default.secret_key
#RUN /usr/share/opensearch/bin/opensearch-plugin install repository-s3

Here is my docker compose:

version: '3'
services:
  opensearch-node1:
    image: thryv/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1
      - cluster.initial_master_nodes=opensearch-node1
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms256m -Xmx256m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_ADMIN_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:2.17.1
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200"]' # must be a string with no spaces when specified as an environment variable
    networks:
      - opensearch-net

volumes:
  opensearch-data1:

networks:
  opensearch-net:

I have containers running, and I can also run API commands using the admin cert (see attached)

part of the opensearch.yml

plugins.security.disabled: false
plugins.security.ssl.transport.pemcert_filepath: opensearch_domains_cert.pem
plugins.security.ssl.transport.pemkey_filepath: opensearch_domains_key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: xxxxx.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: opensearch_domains_cert.pem
plugins.security.ssl.http.pemkey_filepath: opensearch_domains_key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: xxxxx.pem
plugins.security.authcz.admin_dn:
 - "O=blah\\, Inc,OU=Enterprise SSL,CN=yyyyy"
plugins.security.nodes_dn:
 - "O=blah\\, Inc,OU=Enterprise SSL,CN=zzzz"
plugins.security.allow_default_init_securityindex: true
plugins.security.system_indices.enabled: true

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.indices: [".opendistro_security", ".opensearch-observability"]
[opensearch@b003442a204e config]$

How can I troubleshoot this

Thanks in advance

pablo · January 24, 2025, 5:43pm

@stecino I’ve used your docker-compose.yml and Dockerfile

docker-compose.yml

services:
  opensearch-node1:
    container_name: opensearch-node1
    build:
      context: opensearch/
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1
      - cluster.initial_master_nodes=opensearch-node1
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms256m -Xmx256m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=Eliatra123
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:2.17.1
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200"]' # must be a string with no spaces when specified as an environment variable
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
networks:
  opensearch-net:

Dockerfile

FROM opensearchproject/opensearch:2.17.1

COPY opensearch_certs/*.pem /usr/share/opensearch/config/
COPY opensearch.yml /usr/share/opensearch/config/
#COPY securityadmin.sh /usr/share/opensearch/
USER root
RUN yum install procps -y
RUN chown opensearch:opensearch /usr/share/opensearch/config/*.pem
RUN chown opensearch:opensearch /usr/share/opensearch/config/opensearch.yml
USER opensearch
RUN /usr/share/opensearch/bin/opensearch-keystore create
RUN echo 'xxxxxxxxxxx' | /usr/share/opensearch/bin/opensearch-keystore add -xf s3.client.default.access_key
RUN echo 'xxxxxxxxxxx' | /usr/share/opensearch/bin/opensearch-keystore add -xf s3.client.default.secret_key
RUN /usr/share/opensearch/bin/opensearch-plugin install -batch repository-s3

s3 plugin has installed successfully inside the container with Dockerfile

[+] Building 0.1s (16/16) FINISHED                                                                                                         docker:default
 => [opensearch-node1 internal] load build definition from Dockerfile                                                                                0.0s
 => => transferring dockerfile: 920B                                                                                                                 0.0s
 => [opensearch-node1 internal] load metadata for docker.io/opensearchproject/opensearch:2.17.1                                                      0.0s
 => [opensearch-node1 internal] load .dockerignore                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                      0.0s
 => [opensearch-node1  1/10] FROM docker.io/opensearchproject/opensearch:2.17.1                                                                      0.0s
 => [opensearch-node1 internal] load build context                                                                                                   0.0s
 => => transferring context: 1.33kB                                                                                                                  0.0s
 => CACHED [opensearch-node1  2/10] COPY opensearch_certs/*.pem /usr/share/opensearch/config/                                                        0.0s
 => CACHED [opensearch-node1  3/10] COPY opensearch.yml /usr/share/opensearch/config/                                                                0.0s
 => CACHED [opensearch-node1  4/10] RUN yum install procps -y                                                                                        0.0s
 => CACHED [opensearch-node1  5/10] RUN chown opensearch:opensearch /usr/share/opensearch/config/*.pem                                               0.0s
 => CACHED [opensearch-node1  6/10] RUN chown opensearch:opensearch /usr/share/opensearch/config/opensearch.yml                                      0.0s
 => CACHED [opensearch-node1  7/10] RUN /usr/share/opensearch/bin/opensearch-keystore create                                                         0.0s
 => CACHED [opensearch-node1  8/10] RUN echo 'xxxxxxxxxxxxxx' | /usr/share/opensearch/bin/opensearch-keystore add -xf s3.client.default.acces  0.0s
 => CACHED [opensearch-node1  9/10] RUN echo 'xxxxxxxxxxxxxx' | /usr/share/opensearch/bin/opensearch-keystore add -xf s3.  0.0s
 => CACHED [opensearch-node1 10/10] RUN /usr/share/opensearch/bin/opensearch-plugin install -batch repository-s3                                     0.0s
 => [opensearch-node1] exporting to image                                                                                                            0.0s
 => => exporting layers                                                                                                                              0.0s
 => => writing image sha256:1c2fc2961f0e8c7a5b858e06124dfa11bee7ce8220bfc15125724ac4eb37543a                                                         0.0s
 => => naming to docker.io/library/opensearch-23202-opensearch-node1                                                                                 0.0s
 => [opensearch-node1] resolving provenance for metadata file

[opensearch@8e4f2a607ac2 bin]$ ./opensearch-plugin list
opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-custom-codecs
opensearch-flow-framework
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-security-analytics
opensearch-skills
opensearch-sql
opensearch-system-templates
query-insights
repository-s3

I was also able to reinstall s3 plugin inside the container.

[opensearch@8e4f2a607ac2 bin]$ ./opensearch-plugin install repository-s3
-> Installing repository-s3
-> Downloading repository-s3 from opensearch
[=================================================] 100%??
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.io.FilePermission config#plus read
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.RuntimePermission setContextClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission setDefaultAuthenticator
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission aws.configFile read,write
* java.util.PropertyPermission aws.sharedCredentialsFile read,write
* java.util.PropertyPermission opensearch.allow_insecure_settings read,write
* java.util.PropertyPermission opensearch.path.conf read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed repository-s3 with folder name repository-s3

Maybe the issue is related to your Internet connectivity? Do you use any proxy to access Internet?

stecino · January 25, 2025, 12:24am

Thanks for the reply, no I am not using anything. I am trying to run this on my Mac. Once I deployed everything to linux, I don’t have any issues. I also tried to download that plugin zip file, same issue with zip file install

Topic		Replies	Views
Repository-s3 plugin not installed OpenSearch	2	445	May 27, 2024
Getting error when trying to add or install repository-s3 plugin Open Source Elasticsearch and Kibana	1	645	February 7, 2023
S3-repository doesn't work with private pki? OpenSearch troubleshoot , configure	3	1028	September 9, 2022
Opensearch security plugin with certificate chain Security	10	1761	June 12, 2024
Replacing demo certificates with my organization certificates is not working Security troubleshoot	7	1555	January 12, 2022

Can't install repository-s3 plugin in the docker container

Related topics