Opensearch deployment with OpenSearch Operator - Failure no such index [.opendistro_security] - Solved

Security
1

OpenSearch 2.14.0 in GKE Autopilot

I created self-signed certificate and deploying Opensearch via OpenSearch Operator Cluster Helm Chart, and I have production environment that do not allow to access PODs by command. So, I am not able to run plugins/opensearch-security/tools/securityadmin.sh manually.

In non-production environment, I could deploy Openseach cluster with basic authentication + SAML, but during the deployment I needed to run plugins/opensearch-security/tools/securityadmin.sh script manually to create [.opendistro_security] index.
Command executed in non-production was below:

plugins/opensearch-security/tools/securityadmin.sh -cd config/opensearch-security/  -icl -nhnv -cacert config/cert/root-ca.pem -cert config/cert/admin.pem  -key config/cert/admin-key.pem --accept-red-cluster

How can I configure to Helm Chart or docker image to start automatically without manual run of plugins/opensearch-security/tools/securityadmin.sh ?

When don’t execute command appear below error message.

In BOOTSTRAP side:

[2024-06-26T03:03:14,124][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-06-26T03:03:14,124][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-06-26T03:03:14,124][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-06-26T03:03:14,124][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-06-26T03:03:14,124][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-06-26T03:03:14,124][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-06-26T03:03:14,176][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-bootstrap-0] Not yet initialized (you may need to run securityadmin)
[2024-06-26T03:03:14,178][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-bootstrap-0] Not yet initialized (you may need to run securityadmin)
[2024-06-26T03:03:14,180][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-bootstrap-0] Not yet initialized (you may need to run securityadmin)
[2024-06-26T03:03:14,182][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-bootstrap-0] Not yet initialized (you may need to run securityadmin)
[2024-06-26T03:03:15,387][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-bootstrap-0] Not yet initialized (you may need to run securityadmin)

In MASTER node side

[2024-06-26T03:21:59,519][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-data-master-0] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-06-26T03:21:59,519][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-data-master-0] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-06-26T03:21:59,519][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-data-master-0] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-06-26T03:21:59,519][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-data-master-0] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-06-26T03:21:59,519][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-data-master-0] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-06-26T03:22:09,218][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-data-master-0] Not yet initialized (you may need to run securityadmin)
[2024-06-26T03:22:12,520][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-data-master-0] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-06-26T03:22:12,520][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-data-master-0] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-06-26T03:22:12,520][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-data-master-0] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2024-06-26T03:22:12,520][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-data-master-0] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)

My opensearch.yml file is below

plugins.security.audit.type: internal_opensearch
plugins.security.authcz.admin_dn: 
  - "CN=admin"
  - "OU=opensearch-cluster,CN=admin,"
  - "OU=opensearch-cluster,CN=opensearch-cluster"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.nodes_dn: 
  - "CN=opensearch-cluster"
  - "CN=opensearch-cluster-bootstrap-0"
  - "OU=opensearch-cluster,CN=opensearch-cluster-bootstrap-0"
  - "OU=opensearch-cluster,CN=admin"
  - "OU=opensearch-cluster,CN=opensearch-cluster"
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: cert/opensearch-node.pem
plugins.security.ssl.http.pemkey_filepath: cert/opensearch-node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: cert/root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.ssl.transport.pemcert_filepath: cert/opensearch-node.pem
plugins.security.ssl.transport.pemkey_filepath: cert/opensearch-node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: cert/root-ca.pem
plugins.security.allow_unsafe_democertificates: false
#plugins.security.allow_default_init_securityindex: true
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config",".opendistro-alerting-alert*",".opendistro-anomaly-results*",".opendistro-anomaly-detector*",".opendistro-anomaly-checkpoints",".opendistro-anomaly-detection-state",".opendistro-reports-*",".opendistro-notifications-*",".opendistro-notebooks",".opensearch-observability",".opendistro-asynchronous-search-response*",".replication-metadata-store"]

My values.yaml for OpenSearch cluster is below. I was expecting to have updateJob run securityadmin.sh on each PODs but when put transport/http generate = false, I could see that securityconfig POD never start. So, for non-production I ran manually securityadmin.sh. Maybe my problem is because updateJob POD (=securityconfig POD) never start.

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
...
spec:
  security:
    config:
      updateJob:
        resources:
          requests:
            cpu: 1500m
            memory: 3Gi
          limits:
            cpu: 1500m
            memory: 3Gi
      adminCredentialsSecret:
        name: admin-credentials-secret
    tls:
      transport:
        pernode: false
        generate: false
      http:
        generate: false
2

Problem has been solved creating batch job with few option

{{ if .Values.custom.securityconfig.enabled }}
apiVersion: batch/v1
kind: Job
metadata:
  labels:
    job-name: new-securityconfig-update
  name: new-securityconfig-update
spec:
  backoffLimit: 0
  completionMode: NonIndexed
  completions: 1
  parallelism: 1
  suspend: false
  template:
    metadata:
      labels:
        job-name: new-securityconfig-update
      name: new-securityconfig-update
    spec:
      containers:
        - args:
            - >-
              {{- if .Values.custom.securityconfig.update }}
              ADMIN=/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh;
              chmod +x $ADMIN;
              echo '-----------------------------------------------------------------------';
              echo '      This is CONFIG update only';
              echo '-----------------------------------------------------------------------';
              until curl -k --silent https://{{ .Values.opensearchCluster.clusterName }}.namespace.svc.cluster.local:9200;
                do
                echo 'Waiting to connect to the cluster (https://{{ .Values.opensearchCluster.clusterName }}.namespace.svc.cluster.local:9200)'; 
                sleep 120;
              done;
              count=0;
              until $ADMIN -f config/opensearch-security/config.yml -t config -icl -nhnv -cacert config/cert/root-ca.pem -cert config/cert/admin.pem  -key config/cert/admin-key.pem --accept-red-cluster -h {{ .Values.opensearchCluster.clusterName }}.namespace.svc.cluster.local -p 9200 || (( count++>= 20 ));
                do
                echo 'Updating config.yml '; 
                sleep 20;
              done;
              count=0;
              until $ADMIN -f config/opensearch-security/roles_mapping.yml -t rolesmapping -icl -nhnv -cacert config/cert/root-ca.pem -cert config/cert/admin.pem  -key config/cert/admin-key.pem --accept-red-cluster -h {{ .Values.opensearchCluster.clusterName }}.namespace.svc.cluster.local -p 9200 || (( count++>= 20 ));
                do
                echo 'Updating roles_mapping.yml '; 
                sleep 20;
              done;
              {{- else}}
              ADMIN=/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh;
              chmod +x $ADMIN;
              echo '-----------------------------------------------------------------------';
              echo '           /!\   WARNING  back to initian setting';
              echo '-----------------------------------------------------------------------';
              sleep 120;
              until curl -k --silent https://{{ .Values.opensearchCluster.clusterName }}.namespace.svc.cluster.local:9200;
                do
                echo 'Waiting to connect to the cluster (https://{{ .Values.opensearchCluster.clusterName }}.namespace.svc.cluster.local:9200)'; 
                sleep 120;
              done;
              count=0;
              until $ADMIN -f config/opensearch-security/config.yml -t config -icl -nhnv -cacert config/cert/root-ca.pem -cert config/cert/admin.pem  -key config/cert/admin-key.pem --accept-red-cluster -h {{ .Values.opensearchCluster.clusterName }}.namespace.svc.cluster.local -p 9200 || (( count++>= 20 ));
                do
                echo 'Waiting execution completion '; 
                sleep 20;
              done;
              {{- end }}
          command:
            - /bin/bash
            - '-c'
          image: {{ .Values.opensearchCluster.general.image }}
          imagePullPolicy: Always
          name: updater
          resources:
            limits:
              cpu: 500m
              ephemeral-storage: 1Gi
              memory: 2Gi
            requests:
              cpu: 500m
              ephemeral-storage: 1Gi
              memory: 2Gi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - NET_RAW
            privileged: false
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: >-
                /usr/share/opensearch/config/opensearch-security/action_groups.yml
              name: securityconfig
              readOnly: true
              subPath: action_groups.yml
            - mountPath: /usr/share/opensearch/config/opensearch-security/config.yml
              name: securityconfig
              readOnly: true
              subPath: config.yml
            - mountPath: >-
                /usr/share/opensearch/config/opensearch-security/internal_users.yml
              name: securityconfig
              readOnly: true
              subPath: internal_users.yml
            - mountPath: /usr/share/opensearch/config/opensearch-security/nodes_dn.yml
              name: securityconfig
              readOnly: true
              subPath: nodes_dn.yml
            - mountPath: /usr/share/opensearch/config/opensearch-security/roles.yml
              name: securityconfig
              readOnly: true
              subPath: roles.yml
            - mountPath: >-
                /usr/share/opensearch/config/opensearch-security/roles_mapping.yml
              name: securityconfig
              readOnly: true
              subPath: roles_mapping.yml
            - mountPath: /usr/share/opensearch/config/opensearch-security/tenants.yml
              name: securityconfig
              readOnly: true
              subPath: tenants.yml
            - mountPath: /usr/share/opensearch/config/opensearch-security/whitelist.yml
              name: securityconfig
              readOnly: true
              subPath: whitelist.yml
      dnsPolicy: ClusterFirst
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
      terminationGracePeriodSeconds: 5
      tolerations:
        - effect: NoSchedule
          key: kubernetes.io/arch
          operator: Equal
          value: amd64
      volumes:
        - name: securityconfig
          secret:
            defaultMode: 420
            secretName: securityconfig-secret
{{- end }}

And in Values.YML

custom:
  securityconfig:
    #-------------------------------------------------------------------------------------------
    # https://opensearch.org/docs/2.14/security/configuration/security-admin/
    # Running securityadmin.sh overwrites one or more portions of the .opendistro_security index. 
    # Run it with extreme care to avoid losing your existing resources.
    # To avoid this situation, back up your current configuration before making changes and re-running the script:
    #-------------------------------------------------------------------------------------------
    enabled: true
    #############################################################################################
    #/!\ ATTENTION putting update = false will erase setting and push blank initial setting
    # update = true only update security config.yml and roles_mapping.yml files. 
    #  It is require to update SAML settings for example
    #############################################################################################
    update: true
1 Like