Unable to read the file ca.crt. Please make sure this files exists and is readable regarding to permissions

Ghata · February 24, 2025, 9:00am

Hello everyone, I’ve deployed Opensearch with custom SecurityConfig but opensearch goes into crashloopbackoff error:
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[Unable to read the file ca.crt. Please make sure this files exists and is readable regarding to permissions]

To Reproduce
Steps to reproduce the behavior:

Add a custom config.yml via securityConfig.configSecret
Mount custom certificate via secretMounts
Apply config opensearch.yml
Set DISABLE_INSTALL_DEMO_CONFIG to false so that it picks up the opensearch.yml config.
See the error.

Expected behavior
Opensearch runs without errors.

Chart Name
opensearch

Host/Environment (please complete the following information):
opensearch : 2.18.0
opensearch helm chart: 2.31.0

Additional context
Here’s my values.yaml:

replicas: 1
singleNode: true
rbac:
create: true
automountServiceAccountToken: true
persistence:
enabled: false
size: 5Gi
image: “busybox”
imageTag: 1.27.2-buildx
enableInitChown: false
config:
opensearch.yml: |
plugins:
security:
allow_default_init_securityindex: true
ssl:
transport:
pemcert_filepath: tls.crt
pemkey_filepath: tls.key
pemtrustedcas_filepath: ca.crt
http:
enabled: false
system_indices:
enabled: true
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
extraEnvs:

name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
value: “redacted”

name: DISABLE_INSTALL_DEMO_CONFIG
value: ‘false’

name: DISABLE_PERFORMANCE_ANALYZER_AGENT_CLI
value: ‘true’
secretMounts:

name: poc-dashboard-secret
secretName: poc-dashboard-secret
path: /usr/opensearch/config/
plugins:
enabled: true
installList:

repository-s3
securityConfig:
enabled: true
path: /usr/share/opensearch/config/opensearch-security
config:
securityConfigSecret: securityconfig-fdn-dev

poc-dashboard-secret:

apiVersion: v1
kind: Secret
metadata:
name: poc-dashboard-secret
type: kubernetes.io/tls
data:
ca.crt:
tls.key:
tls.crt:

securityconfig-fdn-dev secret:

apiVersion: v1
kind: Secret
metadata:
name: securityconfig-fdn-dev
type: Opaque
stringData:
roles.yml: |-
_meta:
type: “roles”
config_version: 2
test-role:
reserved: true
hidden: false
cluster_permissions:
- “cluster:admin/opendistro/alerting/alerts/ack”
- “cluster:admin/opendistro/alerting/alerts/get”
- “cluster:monitor/"
- "indices:data/read/”
index_permissions:
- index_patterns:
- “"
allowed_actions:
- “indices:admin/template/get”
- “indices:admin/template/put”
- "indices:data/read/”
- “indices:monitor/*”
- “indices:admin/get”
- “indices:admin/mappings/get”
- “indices:admin/aliases/get”
roles_mapping.yml: |-
_meta:
type: “rolesmapping”
config_version: 2

    # Define your roles mapping here

    ## Demo roles mapping

    all_access:
      reserved: false
      users:
      - realm-admin
      backend_roles:
      - "admin"
      description: "Maps admin to all_access"
    
    test-role:
      reserved: false
      hidden: false
      hosts: []
      users:
      - "*"

    own_index:
      reserved: false
      users:
      - "*"
      description: "Allow full access to an index named like the username"

    logstash:
      reserved: false
      backend_roles:
      - "logstash"

    kibana_user:
      reserved: false
      backend_roles:
      - "kibanauser"
      description: "Maps kibanauser to kibana_user"
    
    readall:
      reserved: false
      users:
       - "*"
      backend_roles:
      - "readall"
      
    manage_snapshots:
      reserved: false
      backend_roles:
      - "snapshotrestore"

    kibana_server:
      reserved: true
      users:
      - "kibanaserver"
  config.yml: |-
    _meta:
      type: "config"
      config_version: 2
    config:
      dynamic:
            do_not_fail_on_forbidden: true
            http:
              anonymous_auth_enabled: false
            authc:
              basic_internal_auth_domain:
                description: "Authenticate via HTTP Basic"
                http_enabled: true
                transport_enabled: true
                order: 0
                http_authenticator:
                  type: "basic"
                  challenge: false
                authentication_backend:
                  type: "internal"

              openid_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: 1
                http_authenticator:
                  type: openid
                  challenge: true
                  config:
                    openid_connect_idp:
                      enable_ssl: true
                      verify_hostnames: false
                      # pemtrustedcas_filepath: "/usr/share/opensearch/config/ca.crt"
                      pemtrustedcas_content: |
                          -----BEGIN CERTIFICATE-----
                          redacted
                          -----END CERTIFICATE-----
                    subject_key: preferred_username
                    roles_key: roles
                    openid_connect_url: "https://<keycloakurl>/auth/realms/alpha/.well-known/openid-configuration"
                authentication_backend:
                  type: noop

Anthony · February 25, 2025, 12:39pm

@Ghata can you reformat the config, as its very difficult to read. Surround the config with ``` for better visibility.

The issue you are seeing is the ca file is not being read in. Can you provide config where you set up the secrets and how they are being consumed in values.yaml.

Ghata · February 25, 2025, 1:17pm

Hi Anthony, here’s the requested configuration for cert paths and secrets:

config:
            opensearch.yml: |
              plugins:
                security:
                  allow_default_init_securityindex: true
                  ssl:
                    transport:
                      pemcert_filepath: /etc/opensearch/tls.crt
                      pemkey_filepath: /etc/opensearch/tls.key
                      pemtrustedcas_filepath: /etc/opensearch/ca.crt
                      enforce_hostname_verification: false
                    http:
                      enabled: true
                      pemcert_filepath: /etc/opensearch/tls.crt
                      pemkey_filepath: /etc/opensearch/tls.key
                      pemtrustedcas_filepath: /etc/opensearch/ca.crt
                  system_indices:
                    enabled: true
secretMounts:
            - name: poc-dashboard-secret
              secretName: poc-dashboard-secret
              path: /etc/opensearch/
securityConfig:
            enabled: true
            path: /usr/share/opensearch/config/opensearch-security
            config:
              securityConfigSecret: securityconfig-fdn-dev

With above configuration, error has changed to
java.security.AccessControlException: access denied (“java.io.FilePermission” “/etc/opensearch/ca.crt” “read”

Anthony · February 25, 2025, 3:11pm

This could be related to a permission to access that path, can you try to change the path for certificates to /usr/share/opensearch/config.

If you are still seeing the same issue, can you provide your full values.yaml file, don’t forget to redact any sensitive details.

Ghata · February 26, 2025, 5:46am

Hi Anthony, changing the cert path throws a diff error. Now, Pod is not coming up:

Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/6851bc79-2c29-49f4-9194-68b8e9aa317c/volume-subpaths/config-emptydir/opensearch/2" to rootfs at "/usr/share/opensearch/config/opensearch.yml": mount /var/lib/kubelet/pods/6851bc79-2c29-49f4-9194-68b8e9aa317c/volume-subpaths/config-emptydir/opensearch/2:/usr/share/opensearch/config/opensearch.yml (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown

here’s my values.yaml

image:
  repository: "opensearchproject/opensearch"
imagePullSecrets:
  - name: imagePullSecretValue
replicas: 1
# singleNode: true
rbac:
  create: true
  automountServiceAccountToken: true
persistence:
  enabled: false
  size: 5Gi
  image: "busybox"
  imageTag: 1.27.2-buildx
  enableInitChown: false
config:
  opensearch.yml: |
    plugins:
      security:
        allow_default_init_securityindex: true
        ssl:
          transport:
            pemcert_filepath: /usr/share/opensearch/config/tls.crt
            pemkey_filepath: /usr/share/opensearch/config/tls.key
            pemtrustedcas_filepath: /usr/share/opensearch/config/ca.crt
            enforce_hostname_verification: false
          http:
            enabled: true
            pemcert_filepath: /usr/share/opensearch/config/tls.crt
            pemkey_filepath: /usr/share/opensearch/config/tls.key
            pemtrustedcas_filepath: /usr/share/opensearch/config/ca.crt
        system_indices:
          enabled: true
securityContext:
  allowPrivilegeEscalation: false
  runAsNonRoot: true
extraEnvs: 
  - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
    value: "redacted"
  - name: DISABLE_INSTALL_DEMO_CONFIG
    value: 'true'
  - name: DISABLE_PERFORMANCE_ANALYZER_AGENT_CLI
    value: 'true'
secretMounts:
  - name: poc-dashboard-secret
    secretName: poc-dashboard-secret
    path: /usr/share/opensearch/config/
plugins:
  enabled: true
  installList:
  - repository-s3
securityConfig:
  enabled: true
  path: /usr/share/opensearch/config/opensearch-security
  config:
    securityConfigSecret: securityconfig-fdn-dev

Ghata · February 26, 2025, 3:00pm

Hi @Anthony ,

Any update?

Anthony · February 28, 2025, 2:11pm

@Ghata You should use “subPath” in the secret mounting, as follows:

secretMounts:
  - name: opensearch-certificates
    secretName: opensearch-certificates
    path: /usr/share/opensearch/config/node.pem
    subPath: esnode.pem
  - name: opensearch-certificates-key
    secretName: opensearch-certificates
    path: /usr/share/opensearch/config/key.pem
    subPath: esnode-key.pem
  - name: opensearch-certificates-ca
    secretName: opensearch-certificates
    path: /usr/share/opensearch/config/ca.pem
    subPath: root-ca.pem

In the above case my secrets are named esnode.pem, esnode-key.pem and root-ca.pem, but mounted as files named node.pem, key.pem and ca.pem

Then in your elasticsearch.yml file, only reference these by name:

plugins:
      security:
        ssl:
          transport:
            pemcert_filepath: node.pem
            pemkey_filepath: key.pem
            pemtrustedcas_filepath: ca.pem
            enforce_hostname_verification: false
          http:
            enabled: true
            pemcert_filepath: node.pem
            pemkey_filepath: key.pem
            pemtrustedcas_filepath: ca.pem

Topic		Replies	Views
Failing with read permission on certificates Security	6	1384	December 11, 2024
Caused by: org.opensearch.OpenSearchException: Unable to read the file root-ca.pem. Please make sure this files exists and is readable regarding to permissions OpenSearch configure , install , security-issue	11	103	March 24, 2025
OpenSearch ssl exception (tar) OpenDistro	2	7805	December 9, 2021
Opensearch Exception: No SSL configuration found Security troubleshoot , configure , install	5	185	March 5, 2025
Opensearch in get error failed to load plugin class OpenSearch	3	419	November 22, 2024

Unable to read the file ca.crt. Please make sure this files exists and is readable regarding to permissions

Related topics