How to configure OIDC via the operator?

Security
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch Operator - opensearch-operator-2.4.0

OpenSearch 2.11.1
OS: ubuntu 22.04
Kubernetes: KinD 1.28

Describe the issue:

I am trying to integrate OpenID Connect into OpenSearch via the operator. The dashboard’s oidc integration works (I can see that SSO is succeeding and it’s requesting an id_token in my idp logs. Now I’m trying to configure the backend to accept the tokens but I can’t get masters or nodes to start because they don’t recognize the configuration options via environment variable (see logs section). I feel like I’m missing something very simple.

Configuration:

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: local
  namespace: opensearch-cp
spec:
  general:
    version: "2.11.1"
    httpPort: 9200
    vendor: opensearch
    serviceName: my-cluster
    monitoring:
     enable: true
    pluginsList: ["repository-s3"]
    additionalConfig:
      node.store.allow_mmap: "false"
      openid_auth_domain.http_enabled: "true"
      openid_auth_domain.transport_enabled: "true"
      openid_auth_domain.order: "0"
      openid_auth_domain.http_authenticator.type: openid
      openid_auth_domain.http_authenticator.challenge: "false"
      openid_auth_domain.http_authenticator.config.openid_connect_idp.enable_ssl: "true"
      openid_auth_domain.http_authenticator.config.openid_connect_idp.verify_hostnames: "true"
      openid_auth_domain.http_authenticator.config.openid_connect_idp.pemtrustedcas_filepath: /etc/enterprise-ca/ca.crt
      openid_auth_domain.http_authenticator.config.subject_key: sub
      openid_auth_domain.http_authenticator.config.roles_key: roles
      openid_auth_domain.http_authenticator.config.openid_connect_url: https://k8sou.apps.192-168-2-93.nip.io/auth/idp/opensearch/.well-known/openid-configuration
      openid_auth_domain.authentication_backend.type: noop
    additionalVolumes:
    - name: enterprise-ca
      path: /etc/enterprise-ca
      configMap:
        name: enterprise-root-ca.crt
  dashboards:
    version: "2.11.1"
    enable: true
    replicas: 1
    resources:
      requests:
         memory: "1Gi"
         cpu: "500m"
      limits:
         memory: "1Gi"
         cpu: "500m"
    additionalConfig:
      server.name: opensearch.apps.192-168-2-93.nip.io
      opensearch_security.auth.type: "openid"
      opensearch_security.openid.connect_url: https://k8sou.apps.192-168-2-93.nip.io/auth/idp/opensearch/.well-known/openid-configuration
      opensearch_security.openid.base_redirect_url: https://opensearch.apps.192-168-2-93.nip.io/
      opensearch_security.openid.client_id: opensearch
      opensearch_security.openid.client_secret: BLxvl7oMSx8mxpSGpOV0UtRiKArFLMm4n368nzT4cu9cXuVc3AZq69vcPcQ2zetj
      opensearch_security.openid.scope: openid profile email
      opensearch_security.openid.root_ca: /etc/enterprise-ca/ca.crt
      opensearch_security.openid.refresh_tokens: "true"
    additionalVolumes:
    - name: enterprise-ca
      path: /etc/enterprise-ca
      configMap:
        name: enterprise-root-ca.crt
  confMgmt:
    smartScaler: true
  nodePools:
    - component: masters
      replicas: 3
      diskSize: "3Gi"
      nodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "2Gi"
            cpu: "500m"
      roles:
        - "master"
        - "data"
    - component: nodes
      replicas: 1
      diskSize: "3Gi"
      nodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "2Gi"
            cpu: "500m"
      roles:
        - "data"
    - component: coordinators
      replicas: 1
      diskSize: "3Gi"
      nodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "2Gi"
            cpu: "500m"
      roles:
        - "ingest"

Relevant Logs or Screenshots:

java.lang.IllegalArgumentException: unknown setting [openid_auth_domain.http_authenticator.config.roles_key] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
 at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:608)
 at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:549)
 at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:519)
 at org.opensearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489)
 at org.opensearch.common.settings.SettingsModule.<init>(SettingsModule.java:178)
 at org.opensearch.node.Node.<init>(Node.java:578)
 at org.opensearch.node.Node.<init>(Node.java:407)
 at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
 at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
 at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
 at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
 at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
 at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
 at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
 at org.opensearch.cli.Command.main(Command.java:101)
 at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
 at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
2

I finally figured it out. The configs need to be added to the security section, not the node or master section

1 Like
3

@mlbiam That’s correct.

  security:
    config:
      adminCredentialsSecret:
        name: admin-credentials-secret  # The secret with the admin credentials for the operator to use
      securityConfigSecret:
       name: securityconfig-secret  # The secret containing your customized securityconfig
    tls:
      transport:
        generate: true
      http:
        generate: true
5

@mlbiam Can you post the working configuration for our reference

6

@ar.shashikumar As per @mlbiam comment the below OpenID section must be set in the security section.

You must define a security config as a secret instead of additionalCofnig.
additionalConfig refers to opensearch.yml file.

1 Like
8

@pablo we tried to apply as per the instructions under Security Section. However it fails with an error mentioned below. Does it need to be applied under general: ? Please advise also it would be great if you could review the code as it shows unknown setting for all OIDC parameters

Provided our yaml file for reference

error: error validating "opensearch-cluster.yaml": error validating data: ValidationError(OpenSearchCluster.spec.security): unknown field "additionalConfig" in io.opster.opensearch.v1.OpenSearchCluster.spec.security; if you choose to ignore these errors, turn validation off with --validate=false

Config:

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: opensearch
  namespace: demo
spec:
  security:
    config:
      adminCredentialsSecret:
        name: admin-credentials-secret  # The secret with the admin credentials for the operator to use
      securityConfigSecret:
       name: securityconfig-secret  # The secret containing your customized securityconfig
    tls:
      http:
        generate: true
      transport:
        generate: true
        perNode: true
    additionalConfig:
      node.store.allow_mmap: "false"
      openid_auth_domain.http_enabled: "true"
      openid_auth_domain.transport_enabled: "true"
      openid_auth_domain.order: "0"
      openid_auth_domain.http_authenticator.type: openid
      openid_auth_domain.http_authenticator.challenge: "false"
      openid_auth_domain.http_authenticator.config.openid_connect_idp.enable_ssl: "true"
      openid_auth_domain.http_authenticator.config.openid_connect_idp.verify_hostnames: "true"
      openid_auth_domain.http_authenticator.config.openid_connect_idp.pemtrustedcas_filepath: /etc/cachain/cachain
      openid_auth_domain.http_authenticator.config.subject_key: email 
      openid_auth_domain.http_authenticator.config.roles_key: roles
      openid_auth_domain.http_authenticator.config.openid_connect_url: https://oidc-link/discovery/v2.0/.well-known/openid-configuration
      openid_auth_domain.authentication_backend.type: noop
      basic_internal_auth_domain.http_enabled: "true"
      basic_internal_auth_domain.transport_enabled: "true"
      basic_internal_auth_domain.order: "1"
      basic_internal_auth_domain.http_authenticator.type: basic
      basic_internal_auth_domain.challenge: "false"
      basic_internal_auth_domain.authentication_backend.type: internal

We also tried adding OIDC under general, however we are getting below errors. This shows unknown settings for all openid_auth parameters

        Suppressed: java.lang.IllegalArgumentException: unknown setting [openid_auth_domain.order] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

Configuration

  general:
    setVMMaxMapCount: false
    httpPort: 9200
    serviceName: opensearch
    version: 2.14.0
    pluginsList: ["repository-s3"]
    drainDataNodes: true
    additionalVolumes:
    - name: cachain
      path: /etc/cachain
      secret:
        secretName: cachain
    additionalConfig:
      node.store.allow_mmap: "false"
      openid_auth_domain.http_enabled: "true"
      openid_auth_domain.transport_enabled: "true"
      openid_auth_domain.order: "0"
      openid_auth_domain.http_authenticator.type: openid
      openid_auth_domain.http_authenticator.challenge: "false"
      openid_auth_domain.http_authenticator.config.openid_connect_idp.enable_ssl: "true"
      openid_auth_domain.http_authenticator.config.openid_connect_idp.verify_hostnames: "true"
      openid_auth_domain.http_authenticator.config.openid_connect_idp.pemtrustedcas_filepath: /etc/cachain/cachain
      openid_auth_domain.http_authenticator.config.subject_key: email #need to define email or anyother
      openid_auth_domain.http_authenticator.config.roles_key: roles
      openid_auth_domain.http_authenticator.config.openid_connect_url: https://login.microsoftonline.com/46c98d88-e344-4ed4-8496-4ed7712e255d/discovery/v2.0/.well-known/openid-configuration
      openid_auth_domain.authentication_backend.type: noop
      basic_internal_auth_domain.http_enabled: "true"
      basic_internal_auth_domain.transport_enabled: "true"
      basic_internal_auth_domain.order: "1"
      basic_internal_auth_domain.http_authenticator.type: basic
      basic_internal_auth_domain.challenge: "false"
      basic_internal_auth_domain.authentication_backend.type: internal

As a next testcase, we will configure it under each nodePools and validate as per the document link

9

This test case configuring additionalConfig on each nodePools also having same errors with respect to unknown setting

[2024-11-07T06:05:39,013][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-masters-0] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [openid_auth_domain.http_authenticator.config.roles_key] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:185) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.14.0.jar:2.14.0]
        at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.14.0.jar:2.14.0]
Caused by: java.lang.IllegalArgumentException: unknown setting [openid_auth_domain.http_authenticator.config.roles_key] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
10

@ar.shashikumar The configuration you’re trying to apply is part of the security plugin. The security plugin uses a config.yml file to apply the configuration instead of opensearch.yml

All the actions that you’ve performed will apply the OpenSearch security configuration to the opensearch.yml file.

If you want to use a custom security configuration (e.g., OpenID), you must create a secret with all the security plugin configurations and apply with the previously mentioned option.

You can find an example of such a secret in the OpenSearch Operator documentation

The OpenID configuration requires OpenSearch and OpenSearch Dashboards configuration.
The OpenSearch Dashboards doesn’t use separate config files for security configuration. All of the required security configurations must be set in the opensearch_dashboards.yml.

11

@ar.shashikumar You can always verify the security configuration in the running cluster by running the following API call.

curl --insecure -u admin:<password> https://<OpenSearch_FQDN_or_IP>:9200/_plugins/_security/api/securityconfig?pretty
12 
kns exec -it opensearch-masters-0 -- curl -k -XGET "https://opensearch:9200/_plugins/_security/api/securityconfig?pretty" -u admin
Enter host password for user 'admin':
{
  "config" : {
    "dynamic" : {
      "filtered_alias_mode" : "warn",
      "disable_rest_auth" : false,
      "disable_intertransport_auth" : false,
      "respect_request_indices_options" : false,
      "kibana" : {
        "multitenancy_enabled" : true,
        "private_tenant_enabled" : true,
        "default_tenant" : "",
        "server_username" : "kibanaserver",
        "index" : ".kibana",
        "sign_in_options" : [
          "BASIC"
        ]
      },
      "http" : {
        "anonymous_auth_enabled" : false,
        "xff" : {
          "enabled" : false,
          "internalProxies" : "10\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|192\\.168\\.\\d{1,3}\\.\\d{1,3}|169\\.254\\.\\d{1,3}\\.\\d{1,3}|127\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|172\\.1[6-9]{1}\\.\\d{1,3}\\.\\d{1,3}|172\\.2[0-9]{1}\\.\\d{1,3}\\.\\d{1,3}|172\\.3[0-1]{1}\\.\\d{1,3}\\.\\d{1,3}",
          "remoteIpHeader" : "X-Forwarded-For"
        }
      },
      "authc" : {
        "basic_internal_auth_domain" : {
          "http_enabled" : true,
          "order" : 4,
          "http_authenticator" : {
            "challenge" : true,
            "type" : "basic",
            "config" : { }
          },
          "authentication_backend" : {
            "type" : "intern",
            "config" : { }
          }
        }
      },
      "authz" : { },
      "auth_failure_listeners" : { },
      "do_not_fail_on_forbidden" : false,
      "multi_rolespan_enabled" : true,
      "hosts_resolver_mode" : "ip-only",
      "do_not_fail_on_forbidden_empty" : false,
      "on_behalf_of" : {
        "enabled" : false
      }
    }
  }
}
13

@ar.shashikumar Did you follow my previous suggestions? This output clearly shows that OpenID configuration has not been applied to the OpenSearch security plugin.

14

@pablo Thanks a lot for the pointer. ODIC configuration to be made in securityconfig-secret.yaml and below is the configuration. with the below configuration pods are not getting into running state.

          config.yml: |-
        _meta:
          type: "config"
          config_version: "2"
        config:
          dynamic:
            http:
              anonymous_auth_enabled: false
            authc:
              node.store.allow_mmap: "false"
              basic_internal_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: "1"
                http_authenticator:
                  type: basic
                  challenge: false
                authentication_backend:
                  type: internal
            openid_auth_domain:
              http_enabled: true
              transport_enabled: true
              order: 0
              http_authenticator:
                type: openid
                challenge: false
                config:
                  subject_key: email
                  roles_key: roles
                  openid_connect_idp.enable_ssl: "true"
                  openid_connect_idp.verify_hostnames: "true"
                  auth.multiple_auth_enabled: true
                  openid_connect_idp:
                    pemtrustedcas_filepath: /etc/cachain/cachain
                  auth.type: ["openid", "basicauth"]
                  openid_connect_url: "https://login.microsoftonline.com/4-8496-4ed77/discovery/v2.0/.well-known/openid-configuration"
                  openid.client_id:
                  openid.client_secret:
                  logging.verbose: true
              authentication_backend:
                type: noop```

Error of opensearch-masters-0

[2024-11-07T19:42:00,085][ERROR][o.o.t.n.s.SecureNetty4Transport] [opensearch-masters-0] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378) ~[?:?]
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316) ~[?:?]
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:134) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]


Noticed, Post securityconfig-secret.yaml update, opensearch pods did not automatically performed rollout restart.
15

@ar.shashikumar This is a handshake error on the Transport layer that is done before the security plugin configuration is applied.

Did you follow the security config secret example? You must provide all of the files (roles, roles mapping, tenants etc)

16

@pablo Post deployment of OIDC configuration, the master pod throws an error.

  1. file permission issue: full permissions looks to be applied which was mounted as secret
kns exec -it opensearch-masters-0 -- ls -lrt /etc/cachain/cachain
lrwxrwxrwx 1 root root 14 Nov  8 05:21 /etc/cachain/cachain -> ..data/cachain
kns exec -it opensearch-dashboards-677c6db4f4-tmbdq -- ls -lrt /etc/cachain/cachain
lrwxrwxrwx 1 root root 14 Nov  8 06:17 /etc/cachain/cachain -> ..data/cachain
  1. Unable to initialize auth domain openid_auth_domain: Any suggestions on this error?
[2024-11-08T09:05:59,375][WARN ][o.o.s.s.ReflectionHelper ] [opensearch-masters-0] Unable to enable 'com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator' due to java.lang.reflect.InvocationTargetException
[2024-11-08T09:05:59,376][ERROR][o.o.s.s.DynamicConfigModelV7] [opensearch-masters-0] Unable to initialize auth domain openid_auth_domain=AuthcDomain [http_enabled=true, order=0, http_authenticator=HttpAuthenticator [challenge=false, type=openid, config={subject_key=email, roles_key=roles, openid_connect_idp={enable_ssl=true, verify_hostnames=true, pemtrustedcas_filepath=/etc/cachain/cachain}, openid_connect_url=https://login.microsoftonline.com/46c98496-4ed7712e255d/discovery/v2.0/.well-known/openid-configuration, openid_client_id=c0a11f4cbf20fb3, openid_client_secret=uHN8QqIx6dnE, logging.verbose=true}], authentication_backend=AuthcBackend [type=noop, config={}], description=null] due to OpenSearchException[java.lang.reflect.InvocationTargetException]; nested: InvocationTargetException; nested: RuntimeException[java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/cachain/cachain" "read")]; nested: AccessControlException[access denied ("java.io.FilePermission" "/etc/cachain/cachain" "read")];
org.opensearch.OpenSearchException: java.lang.reflect.InvocationTargetException
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:73) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.lambda$newInstance$1(DynamicConfigModelV7.java:432) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:430) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:329) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.<init>(DynamicConfigModelV7.java:102) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:285) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:570) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.notifyConfigurationListeners(ConfigurationRepository.java:559) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:554) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.loadConfigurationWithLock(ConfigurationRepository.java:538) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:531) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:522) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.action.configupdate.TransportConfigUpdateAction.nodeOperation(TransportConfigUpdateAction.java:128) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.action.configupdate.TransportConfigUpdateAction.nodeOperation(TransportConfigUpdateAction.java:52) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:200) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:328) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:324) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:207) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:317) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:155) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$1.messageReceived(OpenSearchSecurityPlugin.java:841) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:114) [opensearch-index-management-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) [opensearch-performance-analyzer-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.transport.NativeMessageHandler$RequestHandler.doRun(NativeMessageHandler.java:480) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.14.0.jar:2.14.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        ... 30 more
Caused by: java.lang.RuntimeException: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/cachain/cachain" "read")
        at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:93) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        ... 30 more
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/cachain/cachain" "read")
        at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
        at java.base/java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
        at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
        at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:742) ~[?:?]
        at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:789) ~[?:?]
        at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:49) ~[?:?]
        at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171) ~[?:?]
        at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
        at java.base/java.nio.file.spi.FileSystemProvider.readAttributesIfExists(FileSystemProvider.java:1270) ~[?:?]
        at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributesIfExists(UnixFileSystemProvider.java:191) ~[?:?]
        at java.base/java.nio.file.Files.isDirectory(Files.java:2319) ~[?:?]
        at org.opensearch.security.support.PemKeyReader.checkPath(PemKeyReader.java:214) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.support.PemKeyReader.resolve(PemKeyReader.java:290) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.support.PemKeyReader.resolve(PemKeyReader.java:276) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromPem(SettingsBasedSSLConfigurator.java:244) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.configureWithSettings(SettingsBasedSSLConfigurator.java:194) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLContext(SettingsBasedSSLConfigurator.java:117) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLConfig(SettingsBasedSSLConfigurator.java:131) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.getSSLConfig(HTTPJwtKeyByOpenIdConnectAuthenticator.java:65) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.initKeyProvider(HTTPJwtKeyByOpenIdConnectAuthenticator.java:47) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:88) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        ... 30 more

Secrets created manually or by opensearch deployment:

 kns get secret
NAME                         TYPE                DATA   AGE
admin-credentials-secret     Opaque              2      2d11h
cachain                      Opaque              1      2d11h
dashboards-credentials       Opaque              2      2d11h
opensearch-admin-cert        kubernetes.io/tls   3      21m
opensearch-admin-password    Opaque              2      21m
opensearch-ca                Opaque              2      27m
opensearch-http-cert         kubernetes.io/tls   3      22m
opensearch-transport-cert    Opaque              17     22m
securityconfig-secret        Opaque              8      28m

Here is my configuration of both securityconfig-secret.yaml and opensearch-cluster.yaml for reference.

Can you review and advise on the errors mentioned above please.

#cat opensearch-cluster.yaml
#Minimal configuration of a cluster with version 2.X of the operator.
#Note the replacement of 'master' role with 'cluster_manager' on line 49
apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: opensearch
  namespace: demo
spec:
  security:
    config:
      adminCredentialsSecret:
        name: admin-credentials-secret  # The secret with the admin credentials for the operator to use
      securityConfigSecret:
       name: securityconfig-secret  # The secret containing your customized securityconfig
    tls:
      http:
        generate: true
      transport:
        generate: true
        perNode: true
  general:
    setVMMaxMapCount: false
    httpPort: 9200
    serviceName: opensearch
    version: 2.14.0
    pluginsList: ["repository-s3"]
    drainDataNodes: true
    additionalVolumes:
    - name: cachain
      path: /etc/cachain
      secret:
        secretName: cachain
  dashboards:
    additionalVolumes:
    - name: cachain
      path: /etc/cachain
      secret:
        secretName: cachain
    opensearchCredentialsSecret:
      name: dashboards-credentials  # This is the name of your secret that contains the credentials for Dashboards to use
    annotations:
      traffic.sidecar.istio.io/includeInboundPorts: "*"
      traffic.sidecar.istio.io/excludeOutboundPorts: "9200,9300"
      traffic.sidecar.istio.io/excludeInboundPorts: "9200,9300"
    tls:
      enable: false
      generate: true
    version: 2.14.0
    enable: true
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  confMgmt:
    smartScaler: true
  nodePools:
    - component: masters
      pdb:
        enable: true
        minAvailable: 2
      replicas: 3
      jvm: -Xmx2G -Xms2G
      diskSize: "30Gi"
      resources:
         requests:
            memory: "4Gi"
            cpu: "1000m"
         limits:
            memory: "4Gi"
            cpu: "1000m"
      roles:
        - "data"
        - "cluster_manager"
      annotations:
        traffic.sidecar.istio.io/includeInboundPorts: "*"
        traffic.sidecar.istio.io/excludeOutboundPorts: "9200,9300"
        traffic.sidecar.istio.io/excludeInboundPorts: "9200,9300"
    - component: nodes
      replicas: 3
      jvm: -Xmx2G -Xms2G
      diskSize: "30Gi"
      resources:
         requests:
            memory: "4Gi"
            cpu: "2000m"
         limits:
            memory: "4Gi"
            cpu: "2000m"
      roles:
        - "data"
      annotations:
        traffic.sidecar.istio.io/includeInboundPorts: "*"
        traffic.sidecar.istio.io/excludeOutboundPorts: "9200,9300"
        traffic.sidecar.istio.io/excludeInboundPorts: "9200,9300"
    - component: coordinators
      replicas: 1
      jvm: -Xmx2G -Xms2G
      diskSize: "30Gi"
      resources:
         requests:
            memory: "4Gi"
            cpu: "2000m"
         limits:
            memory: "4Gi"
            cpu: "2000m"
      roles:
        - "ingest"
      annotations:
        traffic.sidecar.istio.io/includeInboundPorts: "*"
        traffic.sidecar.istio.io/excludeOutboundPorts: "9200,9300"
        traffic.sidecar.istio.io/excludeInboundPorts: "9200,9300"

#cat securityconfig-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: securityconfig-secret
type: Opaque
stringData:
      action_groups.yml: |-
         _meta:
           type: "actiongroups"
           config_version: 2
      internal_users.yml: |-
        _meta:
          type: "internalusers"
          config_version: 2
        admin:
          hash: "$2a$12$efBxsNSmM.QYK9smZhPxt.dqGy/vG5Nk.1KTdqeGbYpZFnE1oCv0q"
          reserved: true
          backend_roles:
          - "admin"
          description: "Demo admin user"
        dashboarduser:
          hash: "$2a$12$efBxsNSmM.QYK9smZhPxt.dqGy/vG5Nk.1KTdqeGbYpZFnE1oCv0q"
          reserved: true
          backend_roles:
          - "admin"
          description: "Demo OpenSearch Dashboards user"
      nodes_dn.yml: |-
        _meta:
          type: "nodesdn"
          config_version: 2
      whitelist.yml: |-
        _meta:
          type: "whitelist"
          config_version: 2
      tenants.yml: |-
        _meta:
          type: "tenants"
          config_version: 2
      roles_mapping.yml: |-
        _meta:
          type: "rolesmapping"
          config_version: 2
        all_access:
          reserved: false
          backend_roles:
          - "admin"
          description: "Maps admin to all_access"
        own_index:
          reserved: false
          users:
          - "*"
          description: "Allow full access to an index named like the username"
        readall:
          reserved: false
          backend_roles:
          - "readall"
        manage_snapshots:
          reserved: false
          backend_roles:
          - "snapshotrestore"
        dashboard_server:
          reserved: true
          users:
          - "dashboarduser"
      roles.yml: |-
        _meta:
          type: "roles"
          config_version: 2
        dashboard_read_only:
          reserved: true
        security_rest_api_access:
          reserved: true
        # Allows users to view monitors, destinations and alerts
        alerting_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/get'
            - 'cluster:admin/opendistro/alerting/destination/get'
            - 'cluster:admin/opendistro/alerting/monitor/get'
            - 'cluster:admin/opendistro/alerting/monitor/search'
        # Allows users to view and acknowledge alerts
        alerting_ack_alerts:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/*'
        # Allows users to use all alerting functionality
        alerting_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/alerting/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allow users to read Anomaly Detection detectors and results
        anomaly_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/ad/detector/info'
            - 'cluster:admin/opendistro/ad/detector/search'
            - 'cluster:admin/opendistro/ad/detectors/get'
            - 'cluster:admin/opendistro/ad/result/search'
            - 'cluster:admin/opendistro/ad/tasks/search'
            - 'cluster:admin/opendistro/ad/detector/validate'
            - 'cluster:admin/opendistro/ad/result/topAnomalies'
        # Allows users to use all Anomaly Detection functionality
        anomaly_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/ad/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allows users to read Notebooks
        notebooks_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/list'
            - 'cluster:admin/opendistro/notebooks/get'
        # Allows users to all Notebooks functionality
        notebooks_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/create'
            - 'cluster:admin/opendistro/notebooks/update'
            - 'cluster:admin/opendistro/notebooks/delete'
            - 'cluster:admin/opendistro/notebooks/get'
            - 'cluster:admin/opendistro/notebooks/list'
        # Allows users to read observability objects
        observability_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to all Observability functionality
        observability_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/create'
            - 'cluster:admin/opensearch/observability/update'
            - 'cluster:admin/opensearch/observability/delete'
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to read and download Reports
        reports_instances_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to read and download Reports and Report-definitions
        reports_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to all Reports functionality
        reports_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/create'
            - 'cluster:admin/opendistro/reports/definition/update'
            - 'cluster:admin/opendistro/reports/definition/on_demand'
            - 'cluster:admin/opendistro/reports/definition/delete'
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to use all asynchronous-search functionality
        asynchronous_search_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:data/read/search*'
        # Allows users to read stored asynchronous-search results
        asynchronous_search_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/get'
        # Allows user to use all index_management actions - ism policies, rollups, transforms
        index_management_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/opendistro/ism/*"
            - "cluster:admin/opendistro/rollup/*"
            - "cluster:admin/opendistro/transform/*"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:admin/opensearch/ism/*'
        # Allows users to use all cross cluster replication functionality at leader cluster
        cross_cluster_replication_leader_full_access:
          reserved: true
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/read/plugins/replication/changes"
                - "indices:data/read/plugins/replication/file_chunk"
        # Allows users to use all cross cluster replication functionality at follower cluster
        cross_cluster_replication_follower_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/plugins/replication/autofollow/update"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/write/plugins/replication/changes"
                - "indices:admin/plugins/replication/index/start"
                - "indices:admin/plugins/replication/index/pause"
                - "indices:admin/plugins/replication/index/resume"
                - "indices:admin/plugins/replication/index/stop"
                - "indices:admin/plugins/replication/index/update"
                - "indices:admin/plugins/replication/index/status_check"
      config.yml: |-
        _meta:
          type: "config"
          config_version: "2"
        config:
          dynamic:
            http:
              anonymous_auth_enabled: false
            authc:
              basic_internal_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: "4"
                http_authenticator:
                  type: basic
                  challenge: false
                authentication_backend:
                  type: intern
              openid_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: "0"
                http_authenticator:
                  type: openid
                  challenge: false
                  config:
                    subject_key: email
                    roles_key: roles
                    openid_connect_idp:
                      enable_ssl: "true"
                      verify_hostnames: "true"
                      pemtrustedcas_filepath: /etc/cachain/cachain
                    openid_connect_url: "https://login.microsoftonline.com/46c2e255d/discovery/v2.0/.well-known/openid-configuration"
                    openid_client_id: "c0a11fad-0fb3"
                    openid_client_secret: "uHN8QIx6dnE"
                    logging.verbose: true
                authentication_backend:
                  type: noop
17

@ar.shashikumar You don’t need that line. Azure IdP is signed with an Enterprise RootCA. OpenSearch docker image already has this Root CA in his keystore.

Also, when you attach any configuration files or certificates for security plugin, the owner and the group must by opensearch:opensearch. Giving 777 not always works.

18

@pablo I commented the line pemtrustedcas_filepath: in securityconfig-secret.yaml and redeployed the opensearch. All the pods are failed with crashloopbackoff error and aren’t getting into running state post that.

In case i need to change the ownership of the .pem file, how do I do that?

kns get po
NAME                                     READY   STATUS             RESTARTS       AGE
opensearch-bootstrap-0                   0/1     CrashLoopBackOff   5 (107s ago)   4m42s
opensearch-coordinators-0                0/1     CrashLoopBackOff   5 (97s ago)    4m33s
opensearch-masters-0                     0/1     CrashLoopBackOff   5 (104s ago)   4m42s
opensearch-masters-1                     0/1     CrashLoopBackOff   5 (84s ago)    4m26s
opensearch-masters-2                     0/1     CrashLoopBackOff   5 (85s ago)    4m26s
opensearch-nodes-0                       0/1     CrashLoopBackOff   5 (100s ago)   4m42s
opensearch-nodes-1                       0/1     CrashLoopBackOff   5 (82s ago)    4m36s
opensearch-nodes-2                       0/1     CrashLoopBackOff   5 (89s ago)    4m36s
opensearch-securityconfig-update-2gxfc   1/1     Running            0              4m42s

kns describe po opensearch-bootstrap-0
Name: opensearch-bootstrap-0
Namespace: demo
Priority: 0
Service Account: default
Node: pdx05-c06-bspr002-vm-3/100.64.24.28
Start Time: Mon, 11 Nov 2024 22:10:04 +0530
Labels: opster.io/opensearch-cluster=opensearch
Annotations: banzaicloud.com/last-applied:
UEsDBBQACAAIAAAAAAAAAAAAAAAAAAAAAAAIAAAAb3JpZ2luYWzMVt9v2zYQ/lcGPlOynLiZ67eg6TAMSG3ERVFgCASKOltcKJLgD2dGoP99ICXLlO0MrbyHwQ+myLvvPn5H8u4N1W…
cni.projectcalico.org/containerID: 408b8aae60efde5a4976050b48828fd793e406d1a2c4cfb5a2994f2d65da45db
cni.projectcalico.org/podIP: 100.121.130.84/32
cni.projectcalico.org/podIPs: 100.121.130.84/32
kubernetes.io/psp: global-unrestricted-psp
Status: Running
IP: 100.121.130.84
IPs:
IP: 100.121.130.84
Controlled By: OpenSearchCluster/opensearch
Containers:
opensearch:
Container ID: containerd://88c7064a8fc37d84cec35dcebcc3c91831a756e08acd4c08c25417ff8c63c617
Image: docker.io/opensearchproject/opensearch:2.14.0
Image ID: docker.io/opensearchproject/opensearch@sha256:466a49f379bb8889af29d615475e69b7b990898c6987d28470cd7105df9046ff
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: StartError
Message: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting “/var/lib/kubelet/pods/e8f23104-f48f-4137-b095-9c3c214bd2e1/volume-subpaths/config/opensearch/10” to rootfs at “/usr/share/opensearch/config/opensearch.yml”: mount /var/lib/kubelet/pods/e8f23104-f48f-4137-b095-9c3c214bd2e1/volume-subpaths/config/opensearch/10:/usr/share/opensearch/config/opensearch.yml (via /proc/self/fd/6), flags: 0x5001: not a directory: unknown
Exit Code: 128
Started: Thu, 01 Jan 1970 05:30:00 +0530
Finished: Mon, 11 Nov 2024 22:12:59 +0530
Ready: False
Restart Count: 5
Liveness: tcp-socket :9200 delay=10s timeout=5s period=20s #success=1 #failure=10
Startup: tcp-socket :9200 delay=10s timeout=5s period=20s #success=1 #failure=10
Environment:
cluster.initial_master_nodes: opensearch-bootstrap-0
discovery.seed_hosts: opensearch-discovery
cluster.name: opensearch
network.bind_host: 0.0.0.0
network.publish_host: opensearch-bootstrap-0 (v1:metadata.name)
OPENSEARCH_JAVA_OPTS: -Xmx512M -Xms512M
node.roles: cluster_manager
http.port: 9200
Mounts:
/usr/share/opensearch/config from cachain (ro)
/usr/share/opensearch/config/opensearch-security/action_groups.yml from securityconfig (ro,path=“action_groups.yml”)
/usr/share/opensearch/config/opensearch-security/config.yml from securityconfig (ro,path=“config.yml”)
/usr/share/opensearch/config/opensearch-security/internal_users.yml from securityconfig (ro,path=“internal_users.yml”)
/usr/share/opensearch/config/opensearch-security/nodes_dn.yml from securityconfig (ro,path=“nodes_dn.yml”)
/usr/share/opensearch/config/opensearch-security/roles.yml from securityconfig (ro,path=“roles.yml”)
/usr/share/opensearch/config/opensearch-security/roles_mapping.yml from securityconfig (ro,path=“roles_mapping.yml”)
/usr/share/opensearch/config/opensearch-security/tenants.yml from securityconfig (ro,path=“tenants.yml”)
/usr/share/opensearch/config/opensearch-security/whitelist.yml from securityconfig (ro,path=“whitelist.yml”)
/usr/share/opensearch/config/opensearch.yml from config (rw,path=“opensearch.yml”)
/usr/share/opensearch/config/tls-http from http-cert (rw)
/usr/share/opensearch/config/tls-transport from transport-cert (rw)
/usr/share/opensearch/data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bmbql (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
transport-cert:
Type: Secret (a volume populated by a Secret)
SecretName: opensearch-transport-cert
Optional: false
http-cert:
Type: Secret (a volume populated by a Secret)
SecretName: opensearch-http-cert
Optional: false
securityconfig:
Type: Secret (a volume populated by a Secret)
SecretName: securityconfig-secret
Optional: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: opensearch-config
Optional: false
cachain:
Type: Secret (a volume populated by a Secret)
SecretName: cachain
Optional: false
data:
Type: EmptyDir (a temporary directory that shares a pod’s lifetime)
Medium:
SizeLimit:
kube-api-access-bmbql:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

Normal Scheduled 4m55s default-scheduler Successfully assigned demo/opensearch-bootstrap-0 to pdx05-c06-bspr002-vm-3
Warning Failed 4m13s (x4 over 4m52s) kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting “/var/lib/kubelet/pods/e8f23104-f48f-4137-b095-9c3c214bd2e1/volume-subpaths/config/opensearch/10” to rootfs at “/usr/share/opensearch/config/opensearch.yml”: mount /var/lib/kubelet/pods/e8f23104-f48f-4137-b095-9c3c214bd2e1/volume-subpaths/config/opensearch/10:/usr/share/opensearch/config/opensearch.yml (via /proc/self/fd/6), flags: 0x5001: not a directory: unknown
Warning BackOff 3m39s (x11 over 4m51s) kubelet Back-off restarting failed container
Normal Pulled 3m27s (x5 over 4m53s) kubelet Container image “docker.io/opensearchproject/opensearch:2.14.0” already present on machine
Normal Created 3m27s (x5 over 4m53s) kubelet Created container opensearch

19

I’ve checked your security config secret and OpenSearchCluster manifest. I had no issues with getting these pods in the running state. I had no ChrashLoopBackOff

The reported error regards opensearch.yml instead of the security plugin.
I tested the manifest that was shared a few days ago. Maybe you made some changes recently that caused this error.

20

Sorry for the confusion. Yes you are right. I had made changes on opensearch.yml that contributed to crashloopback error. I reverted back the changes made with respect to additionalVolumes section

21

@ar.shashikumar What is your current status? Did you solve your OpenID issue?

22

@pablo Yes, OIDC is working now. thanks for checking