OpenSearch Dashboards is NOT Deployed after the cluster(master 3, data2) configuration finished

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

  • kubernetes: v1.20.7
  • opensearch-operator: v2.6.0
  • opensearch & opensearch-dashboards: v2.14.0
    (from On-premise Harbor Repository)

Describe the issue:

  • After the configuration of OpenSearchCluster resource (as below manifests), one pod for the dashboards instance is NOT deployed.
  • A job for securityconfig-update is complete so the .opendistro_security index exists. Also, a bootstrap pod had been terminated normally after leader election between Master nodes.
  • I can’t find any error logs or comment about ‘dashboards’ even the below manifest should deploy ONE replica of pod for opensearch-dashboards.

Configuration:

opensearchCluster:
  enabled: true
  general:
    httpPort: "9200"
    version: v2.14.0
    image: harbor-xxx.xxx.com/library/opensearchproject/opensearch:v2.14.0
    serviceName: "test-opensearch-cluster"
    drainDataNodes: true
    setVMMaxMapCount: true 
    # it doesn't work. This option let an InitContainer does "sysctl -w vm.max_map_count=262144", 
    # but `max_map_count` depends on the value of kernel, i.e the Kubernetes Worker Node (where the pods start)
    podSecurityContext:
      runAsUser: 1000
      runAsGroup: 1000
    securityContext:
      allowPrivilegeEscalation: true
      privileged: true
  initHelper:
    image: "harbor-xxx.xxx.com/nexus/docker-mig/library/busybox:1.31.1"
    imagePullPolicy: IfNotPresent
    # For setVMMaxMapCount=true, but this initContainer is useless since the upper comment(about kernel) I wrote.
  dashboards:
    enable: true
    replicas: 1
    version: v2.14.0
    image: harbor-xxx.xxx.com/library/opensearchproject/opensearch-dashboards:v2.14.0
    resources:
      requests:
        memory: "1Gi"
        cpu: "500m"
      limits:
        memory: "1Gi"
        cpu: "500m"
    opensearchCredentialsSecret:
      name: admin-credentials-secret
    tls:
      enable: false
      generate: false
  nodePools:
    - component: master
      replicas: 3
      diskSize: "10Gi"
      persistence:
        pvc:
          storageClass: "sc-nfs-app-retain"
          accessModes:
           - ReadWriteOnce
      roles:
        - "cluster_manager"
        - "master"
      resources:
        requests:
          memory: "4Gi"
          cpu: "1"
        limits:
          memory: "4Gi"
          cpu: "2"
      env:
        - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
          value: "PASSWORD"
    - component: data
      replicas: 2
      diskSize: "100Gi"
      persistence:
        pvc:
          storageClass: "sc-nfs-app-retain"
          accessModes:
           - ReadWriteOnce
      roles:
        - "data"
        - "ingest"
        - "ml"
      resources:
        requests:
          memory: "8Gi"
          cpu: "2"
        limits:
          memory: "8Gi"
          cpu: "4"
      env:
        - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
          value: "PASSWORD"
  security:
    tls:
      transport:
        generate: true
        perNode: true
      http:
        generate: true
    config:
      adminCredentialsSecret:
         name: admin-credentials-secret
      securityConfigSecret:
         name: securityconfig-secret

Relevant Logs or Screenshots:

This is the logs from ArgoCD.

# StatefulSet : test-opensearch-cluster-master
...
[2024-07-05T07:27:59,116][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-2] Running full sweep 
[2024-07-05T07:28:32,508][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-0] Running full sweep 
[2024-07-05T07:31:05,202][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-1] Running full sweep 
[2024-07-05T07:32:59,117][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-2] Running full sweep 
[2024-07-05T07:33:32,509][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-0] Running full sweep 
[2024-07-05T07:34:25,122][INFO ][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [test-opensearch-cluster-master-1] Starting housekeeping task for auto refresh streaming jobs. 
[2024-07-05T07:34:25,123][INFO ][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [test-opensearch-cluster-master-1] Finished housekeeping task for auto refresh streaming jobs. 
[2024-07-05T07:36:05,203][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-1] Running full sweep 
[2024-07-05T07:37:59,117][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-2] Running full sweep 
[2024-07-05T07:38:32,509][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-0] Running full sweep 
[2024-07-05T07:41:05,203][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-1] Running full sweep 
[2024-07-05T07:42:59,118][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-2] Running full sweep 
[2024-07-05T07:43:32,510][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-0] Running full sweep 
[2024-07-05T07:46:05,204][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-1] Running full sweep 
[2024-07-05T07:47:59,118][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-2] Running full sweep 
[2024-07-05T07:48:32,510][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-0] Running full sweep 
[2024-07-05T07:49:25,101][INFO ][o.o.a.c.HourlyCron       ] [test-opensearch-cluster-master-1] Hourly maintenance succeeds 
[2024-07-05T07:49:25,123][INFO ][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [test-opensearch-cluster-master-1] Starting housekeeping task for auto refresh streaming jobs. 
[2024-07-05T07:49:25,123][INFO ][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [test-opensearch-cluster-master-1] Finished housekeeping task for auto refresh streaming jobs. 
[2024-07-05T07:51:05,204][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-1] Running full sweep
# Pod : opensearch-operator-controller-manager-6c44cc9df5-69j4l
...
{"level":"debug","ts":"2024-07-05T07:19:51.212Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"089fa76b-dca0-4780-9e18-289477965d12","name":"test-opensearch-cluster-data","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"error","ts":"2024-07-05T07:19:51.262Z","msg":"Reconciler error","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"089fa76b-dca0-4780-9e18-289477965d12","error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource; failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorCauses":[{"error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorVerbose":"failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource\ngetting resource failed\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).delete\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:651\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:523\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s.K8sClientImpl.ReconcileResource\n\t/workspace/pkg/reconcilers/k8s/client.go:198\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).handlePDB\n\t/workspace/pkg/reconcilers/cluster.go:410\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).reconcileNodeStatefulSet\n\t/workspace/pkg/reconcilers/cluster.go:240\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).Reconcile\n\t/workspace/pkg/reconcilers/cluster.go:111\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to delete resource"},{"error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorVerbose":"failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource\ngetting resource failed\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).delete\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:651\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:523\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s.K8sClientImpl.ReconcileResource\n\t/workspace/pkg/reconcilers/k8s/client.go:198\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).handlePDB\n\t/workspace/pkg/reconcilers/cluster.go:410\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).reconcileNodeStatefulSet\n\t/workspace/pkg/reconcilers/cluster.go:240\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).Reconcile\n\t/workspace/pkg/reconcilers/cluster.go:111\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to delete resource"}],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"info","ts":"2024-07-05T07:36:31.263Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","cluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"}}
{"level":"info","ts":"2024-07-05T07:36:31.276Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","interface":"transport"}
{"level":"info","ts":"2024-07-05T07:36:31.277Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","interface":"http"}
{"level":"debug","ts":"2024-07-05T07:36:31.277Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster-config","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"ConfigMap"}
{"level":"debug","ts":"2024-07-05T07:36:31.278Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"debug","ts":"2024-07-05T07:36:31.279Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster-discovery","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"debug","ts":"2024-07-05T07:36:31.279Z","msg":"resource diff","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster-admin-password","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Secret"}
{"level":"debug","ts":"2024-07-05T07:36:31.280Z","msg":"updating resource","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster-admin-password","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Secret"}
{"level":"debug","ts":"2024-07-05T07:36:31.283Z","msg":"resource updated","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster-admin-password","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Secret"}
{"level":"debug","ts":"2024-07-05T07:36:31.284Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster-master","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"debug","ts":"2024-07-05T07:36:31.286Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster-data","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"error","ts":"2024-07-05T07:36:31.316Z","msg":"Reconciler error","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource; failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorCauses":[{"error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorVerbose":"failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource\ngetting resource failed\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).delete\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:651\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:523\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s.K8sClientImpl.ReconcileResource\n\t/workspace/pkg/reconcilers/k8s/client.go:198\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).handlePDB\n\t/workspace/pkg/reconcilers/cluster.go:410\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).reconcileNodeStatefulSet\n\t/workspace/pkg/reconcilers/cluster.go:240\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).Reconcile\n\t/workspace/pkg/reconcilers/cluster.go:111\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to delete resource"},{"error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorVerbose":"failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource\ngetting resource failed\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).delete\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:651\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:523\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s.K8sClientImpl.ReconcileResource\n\t/workspace/pkg/reconcilers/k8s/client.go:198\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).handlePDB\n\t/workspace/pkg/reconcilers/cluster.go:410\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).reconcileNodeStatefulSet\n\t/workspace/pkg/reconcilers/cluster.go:240\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).Reconcile\n\t/workspace/pkg/reconcilers/cluster.go:111\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to delete resource"}],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"info","ts":"2024-07-05T07:53:11.317Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","cluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"}}
{"level":"info","ts":"2024-07-05T07:53:11.331Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","interface":"transport"}
{"level":"info","ts":"2024-07-05T07:53:11.331Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","interface":"http"}
{"level":"debug","ts":"2024-07-05T07:53:11.331Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster-config","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"ConfigMap"}
{"level":"debug","ts":"2024-07-05T07:53:11.332Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"debug","ts":"2024-07-05T07:53:11.333Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster-discovery","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"debug","ts":"2024-07-05T07:53:11.333Z","msg":"resource diff","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster-admin-password","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Secret"}
{"level":"debug","ts":"2024-07-05T07:53:11.334Z","msg":"updating resource","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster-admin-password","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Secret"}
{"level":"debug","ts":"2024-07-05T07:53:11.338Z","msg":"resource updated","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster-admin-password","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Secret"}
{"level":"debug","ts":"2024-07-05T07:53:11.339Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster-master","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"debug","ts":"2024-07-05T07:53:11.341Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster-data","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"error","ts":"2024-07-05T07:53:11.378Z","msg":"Reconciler error","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource; failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorCauses":[{"error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorVerbose":"failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource\ngetting resource failed\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).delete\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:651\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:523\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s.K8sClientImpl.ReconcileResource\n\t/workspace/pkg/reconcilers/k8s/client.go:198\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).handlePDB\n\t/workspace/pkg/reconcilers/cluster.go:410\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).reconcileNodeStatefulSet\n\t/workspace/pkg/reconcilers/cluster.go:240\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).Reconcile\n\t/workspace/pkg/reconcilers/cluster.go:111\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to delete resource"},{"error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorVerbose":"failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource\ngetting resource failed\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).delete\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:651\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:523\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s.K8sClientImpl.ReconcileResource\n\t/workspace/pkg/reconcilers/k8s/client.go:198\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).handlePDB\n\t/workspace/pkg/reconcilers/cluster.go:410\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).reconcileNodeStatefulSet\n\t/workspace/pkg/reconcilers/cluster.go:240\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).Reconcile\n\t/workspace/pkg/reconcilers/cluster.go:111\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to delete resource"}],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}

When I redirect to http://test-opensearch-cluster.srep01.xxx.com/ trying to host On-premise using Ingress resource,

Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206c6f63616c686f73743a393230300d0a557365722d4167656e743a206375726c2f382e352e300d0a4163636570743a202a2f2a0d0a0d0a
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1314) ~[netty-handler-4.1.109.Final.jar:4.1.109.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[netty-handler-4.1.109.Final.jar:4.1.109.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]
	... 16 more

I’ve tested OpenSearch Operator with Kubernetes 1.28, OpenSearch 2.14 and the example from GitHub. The 3 node OpenSearch cluster and OpenSearch Dashboards have been deployed successfully.

hi pablo, how did you set up environment for Kubernetes? I am currently working at On-premise Kubernetes.

Maybe there are differences between your and my environment such as LB(MetalLB for On-premise but AWS ALB for cloud).

Looking into my manifest file, config for tls.enable and tls.generate is FALSE. Does is make a big change?

@yeonghyeonKo My setup is based on virtual machines with Ubuntu 22 (1 master + 2 workers). It is my local server setup.
I don’t use LB as it is not needed for the deployment.

tls.enable and tls.generate are only for server-side encryption of OpenSearch Dashboards.
The OpenSearch Dashboards pod should be deployed no matter the setting.

Regarding the Operator controller pod logs, did you check all the logs?
You need to use the --all-containers flag to see logs from all controller-related pods.