OpenSearch Dashboards is NOT Deployed after the cluster(master 3, data2) configuration finished

OpenSearch
, , ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

  • kubernetes: v1.20.7
  • opensearch-operator: v2.6.0
  • opensearch & opensearch-dashboards: v2.14.0
    (from On-premise Harbor Repository)

Describe the issue:

  • After the configuration of OpenSearchCluster resource (as below manifests), one pod for the dashboards instance is NOT deployed.
  • A job for securityconfig-update is complete so the .opendistro_security index exists. Also, a bootstrap pod had been terminated normally after leader election between Master nodes.
  • I can’t find any error logs or comment about ‘dashboards’ even the below manifest should deploy ONE replica of pod for opensearch-dashboards.

image
image1061×326 13 KB

Configuration:

opensearchCluster:
  enabled: true
  general:
    httpPort: "9200"
    version: v2.14.0
    image: harbor-xxx.xxx.com/library/opensearchproject/opensearch:v2.14.0
    serviceName: "test-opensearch-cluster"
    drainDataNodes: true
    setVMMaxMapCount: true 
    # it doesn't work. This option let an InitContainer does "sysctl -w vm.max_map_count=262144", 
    # but `max_map_count` depends on the value of kernel, i.e the Kubernetes Worker Node (where the pods start)
    podSecurityContext:
      runAsUser: 1000
      runAsGroup: 1000
    securityContext:
      allowPrivilegeEscalation: true
      privileged: true
  initHelper:
    image: "harbor-xxx.xxx.com/nexus/docker-mig/library/busybox:1.31.1"
    imagePullPolicy: IfNotPresent
    # For setVMMaxMapCount=true, but this initContainer is useless since the upper comment(about kernel) I wrote.
  dashboards:
    enable: true
    replicas: 1
    version: v2.14.0
    image: harbor-xxx.xxx.com/library/opensearchproject/opensearch-dashboards:v2.14.0
    resources:
      requests:
        memory: "1Gi"
        cpu: "500m"
      limits:
        memory: "1Gi"
        cpu: "500m"
    opensearchCredentialsSecret:
      name: admin-credentials-secret
    tls:
      enable: false
      generate: false
  nodePools:
    - component: master
      replicas: 3
      diskSize: "10Gi"
      persistence:
        pvc:
          storageClass: "sc-nfs-app-retain"
          accessModes:
           - ReadWriteOnce
      roles:
        - "cluster_manager"
        - "master"
      resources:
        requests:
          memory: "4Gi"
          cpu: "1"
        limits:
          memory: "4Gi"
          cpu: "2"
      env:
        - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
          value: "PASSWORD"
    - component: data
      replicas: 2
      diskSize: "100Gi"
      persistence:
        pvc:
          storageClass: "sc-nfs-app-retain"
          accessModes:
           - ReadWriteOnce
      roles:
        - "data"
        - "ingest"
        - "ml"
      resources:
        requests:
          memory: "8Gi"
          cpu: "2"
        limits:
          memory: "8Gi"
          cpu: "4"
      env:
        - name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
          value: "PASSWORD"
  security:
    tls:
      transport:
        generate: true
        perNode: true
      http:
        generate: true
    config:
      adminCredentialsSecret:
         name: admin-credentials-secret
      securityConfigSecret:
         name: securityconfig-secret

Relevant Logs or Screenshots:

This is the logs from ArgoCD.

# StatefulSet : test-opensearch-cluster-master
...
[2024-07-05T07:27:59,116][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-2] Running full sweep 
[2024-07-05T07:28:32,508][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-0] Running full sweep 
[2024-07-05T07:31:05,202][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-1] Running full sweep 
[2024-07-05T07:32:59,117][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-2] Running full sweep 
[2024-07-05T07:33:32,509][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-0] Running full sweep 
[2024-07-05T07:34:25,122][INFO ][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [test-opensearch-cluster-master-1] Starting housekeeping task for auto refresh streaming jobs. 
[2024-07-05T07:34:25,123][INFO ][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [test-opensearch-cluster-master-1] Finished housekeeping task for auto refresh streaming jobs. 
[2024-07-05T07:36:05,203][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-1] Running full sweep 
[2024-07-05T07:37:59,117][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-2] Running full sweep 
[2024-07-05T07:38:32,509][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-0] Running full sweep 
[2024-07-05T07:41:05,203][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-1] Running full sweep 
[2024-07-05T07:42:59,118][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-2] Running full sweep 
[2024-07-05T07:43:32,510][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-0] Running full sweep 
[2024-07-05T07:46:05,204][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-1] Running full sweep 
[2024-07-05T07:47:59,118][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-2] Running full sweep 
[2024-07-05T07:48:32,510][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-0] Running full sweep 
[2024-07-05T07:49:25,101][INFO ][o.o.a.c.HourlyCron       ] [test-opensearch-cluster-master-1] Hourly maintenance succeeds 
[2024-07-05T07:49:25,123][INFO ][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [test-opensearch-cluster-master-1] Starting housekeeping task for auto refresh streaming jobs. 
[2024-07-05T07:49:25,123][INFO ][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [test-opensearch-cluster-master-1] Finished housekeeping task for auto refresh streaming jobs. 
[2024-07-05T07:51:05,204][INFO ][o.o.j.s.JobSweeper       ] [test-opensearch-cluster-master-1] Running full sweep

# Pod : opensearch-operator-controller-manager-6c44cc9df5-69j4l
...
{"level":"debug","ts":"2024-07-05T07:19:51.212Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"089fa76b-dca0-4780-9e18-289477965d12","name":"test-opensearch-cluster-data","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"error","ts":"2024-07-05T07:19:51.262Z","msg":"Reconciler error","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"089fa76b-dca0-4780-9e18-289477965d12","error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource; failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorCauses":[{"error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorVerbose":"failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource\ngetting resource failed\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).delete\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:651\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:523\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s.K8sClientImpl.ReconcileResource\n\t/workspace/pkg/reconcilers/k8s/client.go:198\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).handlePDB\n\t/workspace/pkg/reconcilers/cluster.go:410\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).reconcileNodeStatefulSet\n\t/workspace/pkg/reconcilers/cluster.go:240\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).Reconcile\n\t/workspace/pkg/reconcilers/cluster.go:111\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to delete resource"},{"error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorVerbose":"failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource\ngetting resource failed\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).delete\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:651\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:523\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s.K8sClientImpl.ReconcileResource\n\t/workspace/pkg/reconcilers/k8s/client.go:198\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).handlePDB\n\t/workspace/pkg/reconcilers/cluster.go:410\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).reconcileNodeStatefulSet\n\t/workspace/pkg/reconcilers/cluster.go:240\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).Reconcile\n\t/workspace/pkg/reconcilers/cluster.go:111\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to delete resource"}],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"info","ts":"2024-07-05T07:36:31.263Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","cluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"}}
{"level":"info","ts":"2024-07-05T07:36:31.276Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","interface":"transport"}
{"level":"info","ts":"2024-07-05T07:36:31.277Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","interface":"http"}
{"level":"debug","ts":"2024-07-05T07:36:31.277Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster-config","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"ConfigMap"}
{"level":"debug","ts":"2024-07-05T07:36:31.278Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"debug","ts":"2024-07-05T07:36:31.279Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster-discovery","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"debug","ts":"2024-07-05T07:36:31.279Z","msg":"resource diff","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster-admin-password","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Secret"}
{"level":"debug","ts":"2024-07-05T07:36:31.280Z","msg":"updating resource","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster-admin-password","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Secret"}
{"level":"debug","ts":"2024-07-05T07:36:31.283Z","msg":"resource updated","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster-admin-password","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Secret"}
{"level":"debug","ts":"2024-07-05T07:36:31.284Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster-master","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"debug","ts":"2024-07-05T07:36:31.286Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","name":"test-opensearch-cluster-data","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"error","ts":"2024-07-05T07:36:31.316Z","msg":"Reconciler error","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"a0edb563-35ee-43af-9a19-dbcc9da7808f","error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource; failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorCauses":[{"error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorVerbose":"failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource\ngetting resource failed\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).delete\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:651\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:523\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s.K8sClientImpl.ReconcileResource\n\t/workspace/pkg/reconcilers/k8s/client.go:198\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).handlePDB\n\t/workspace/pkg/reconcilers/cluster.go:410\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).reconcileNodeStatefulSet\n\t/workspace/pkg/reconcilers/cluster.go:240\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).Reconcile\n\t/workspace/pkg/reconcilers/cluster.go:111\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to delete resource"},{"error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorVerbose":"failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource\ngetting resource failed\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).delete\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:651\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:523\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s.K8sClientImpl.ReconcileResource\n\t/workspace/pkg/reconcilers/k8s/client.go:198\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).handlePDB\n\t/workspace/pkg/reconcilers/cluster.go:410\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).reconcileNodeStatefulSet\n\t/workspace/pkg/reconcilers/cluster.go:240\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).Reconcile\n\t/workspace/pkg/reconcilers/cluster.go:111\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to delete resource"}],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"info","ts":"2024-07-05T07:53:11.317Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","cluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"}}
{"level":"info","ts":"2024-07-05T07:53:11.331Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","interface":"transport"}
{"level":"info","ts":"2024-07-05T07:53:11.331Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","interface":"http"}
{"level":"debug","ts":"2024-07-05T07:53:11.331Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster-config","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"ConfigMap"}
{"level":"debug","ts":"2024-07-05T07:53:11.332Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"debug","ts":"2024-07-05T07:53:11.333Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster-discovery","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"debug","ts":"2024-07-05T07:53:11.333Z","msg":"resource diff","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster-admin-password","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Secret"}
{"level":"debug","ts":"2024-07-05T07:53:11.334Z","msg":"updating resource","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster-admin-password","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Secret"}
{"level":"debug","ts":"2024-07-05T07:53:11.338Z","msg":"resource updated","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster-admin-password","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Secret"}
{"level":"debug","ts":"2024-07-05T07:53:11.339Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster-master","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"debug","ts":"2024-07-05T07:53:11.341Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","name":"test-opensearch-cluster-data","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"error","ts":"2024-07-05T07:53:11.378Z","msg":"Reconciler error","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"cfa91fdf-5e47-4216-b95f-53a26640cbee","error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource; failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorCauses":[{"error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorVerbose":"failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource\ngetting resource failed\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).delete\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:651\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:523\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s.K8sClientImpl.ReconcileResource\n\t/workspace/pkg/reconcilers/k8s/client.go:198\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).handlePDB\n\t/workspace/pkg/reconcilers/cluster.go:410\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).reconcileNodeStatefulSet\n\t/workspace/pkg/reconcilers/cluster.go:240\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).Reconcile\n\t/workspace/pkg/reconcilers/cluster.go:111\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to delete resource"},{"error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorVerbose":"failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource\ngetting resource failed\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).delete\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:651\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:523\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s.K8sClientImpl.ReconcileResource\n\t/workspace/pkg/reconcilers/k8s/client.go:198\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).handlePDB\n\t/workspace/pkg/reconcilers/cluster.go:410\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).reconcileNodeStatefulSet\n\t/workspace/pkg/reconcilers/cluster.go:240\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).Reconcile\n\t/workspace/pkg/reconcilers/cluster.go:111\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to delete resource"}],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}

When I redirect to “http://test-opensearch-cluster.srep01.xxx.com/” trying to host On-premise using Ingress resource,

Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206c6f63616c686f73743a393230300d0a557365722d4167656e743a206375726c2f382e352e300d0a4163636570743a202a2f2a0d0a0d0a
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1314) ~[netty-handler-4.1.109.Final.jar:4.1.109.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[netty-handler-4.1.109.Final.jar:4.1.109.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]
	... 16 more
2

I’ve tested OpenSearch Operator with Kubernetes 1.28, OpenSearch 2.14 and the example from GitHub. The 3 node OpenSearch cluster and OpenSearch Dashboards have been deployed successfully.

3

hi pablo, how did you set up environment for Kubernetes? I am currently working at On-premise Kubernetes.

Maybe there are differences between your and my environment such as LB(MetalLB for On-premise but AWS ALB for cloud).

Looking into my manifest file, config for tls.enable and tls.generate is FALSE. Does is make a big change?

4

@yeonghyeonKo My setup is based on virtual machines with Ubuntu 22 (1 master + 2 workers). It is my local server setup.
I don’t use LB as it is not needed for the deployment.

tls.enable and tls.generate are only for server-side encryption of OpenSearch Dashboards.
The OpenSearch Dashboards pod should be deployed no matter the setting.

Regarding the Operator controller pod logs, did you check all the logs?
You need to use the --all-containers flag to see logs from all controller-related pods.

5

As you said there are two running containers in [opensearch-operator-controller-manager-xxx] pod, kube-rbac-proxy and operator-controller-manager.


  1. Logs for [kube-rbac-proxy] container
  • I can’t find anything wrong in logs for [kube-rbac-proxy] container.
Flag --logtostderr has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components
I0705 06:33:08.713843       1 flags.go:64] FLAG: --add-dir-header="false"
I0705 06:33:08.713929       1 flags.go:64] FLAG: --allow-paths="[]"
I0705 06:33:08.713939       1 flags.go:64] FLAG: --alsologtostderr="false"
I0705 06:33:08.713943       1 flags.go:64] FLAG: --auth-header-fields-enabled="false"
I0705 06:33:08.713949       1 flags.go:64] FLAG: --auth-header-groups-field-name="x-remote-groups"
I0705 06:33:08.713955       1 flags.go:64] FLAG: --auth-header-groups-field-separator="|"
I0705 06:33:08.713959       1 flags.go:64] FLAG: --auth-header-user-field-name="x-remote-user"
I0705 06:33:08.713963       1 flags.go:64] FLAG: --auth-token-audiences="[]"
I0705 06:33:08.713979       1 flags.go:64] FLAG: --client-ca-file=""
I0705 06:33:08.713983       1 flags.go:64] FLAG: --config-file=""
I0705 06:33:08.713987       1 flags.go:64] FLAG: --help="false"
I0705 06:33:08.713991       1 flags.go:64] FLAG: --http2-disable="false"
I0705 06:33:08.713995       1 flags.go:64] FLAG: --http2-max-concurrent-streams="100"
I0705 06:33:08.713999       1 flags.go:64] FLAG: --http2-max-size="262144"
I0705 06:33:08.714003       1 flags.go:64] FLAG: --ignore-paths="[]"
I0705 06:33:08.714010       1 flags.go:64] FLAG: --insecure-listen-address=""
I0705 06:33:08.714014       1 flags.go:64] FLAG: --kubeconfig=""
I0705 06:33:08.714017       1 flags.go:64] FLAG: --log-backtrace-at=":0"
I0705 06:33:08.714032       1 flags.go:64] FLAG: --log-dir=""
I0705 06:33:08.714036       1 flags.go:64] FLAG: --log-file=""
I0705 06:33:08.714043       1 flags.go:64] FLAG: --log-file-max-size="1800"
I0705 06:33:08.714049       1 flags.go:64] FLAG: --log-flush-frequency="5s"
I0705 06:33:08.714052       1 flags.go:64] FLAG: --logtostderr="true"
I0705 06:33:08.714056       1 flags.go:64] FLAG: --oidc-ca-file=""
I0705 06:33:08.714059       1 flags.go:64] FLAG: --oidc-clientID=""
I0705 06:33:08.714063       1 flags.go:64] FLAG: --oidc-groups-claim="groups"
I0705 06:33:08.714067       1 flags.go:64] FLAG: --oidc-groups-prefix=""
I0705 06:33:08.714070       1 flags.go:64] FLAG: --oidc-issuer=""
I0705 06:33:08.714077       1 flags.go:64] FLAG: --oidc-sign-alg="[RS256]"
I0705 06:33:08.714086       1 flags.go:64] FLAG: --oidc-username-claim="email"
I0705 06:33:08.714090       1 flags.go:64] FLAG: --one-output="false"
I0705 06:33:08.714094       1 flags.go:64] FLAG: --proxy-endpoints-port="10443"
I0705 06:33:08.714099       1 flags.go:64] FLAG: --secure-listen-address="0.0.0.0:8443"
I0705 06:33:08.714106       1 flags.go:64] FLAG: --skip-headers="false"
I0705 06:33:08.714110       1 flags.go:64] FLAG: --skip-log-headers="false"
I0705 06:33:08.714113       1 flags.go:64] FLAG: --stderrthreshold="2"
I0705 06:33:08.714117       1 flags.go:64] FLAG: --tls-cert-file=""
I0705 06:33:08.714120       1 flags.go:64] FLAG: --tls-cipher-suites="[]"
I0705 06:33:08.714132       1 flags.go:64] FLAG: --tls-min-version="VersionTLS12"
I0705 06:33:08.714136       1 flags.go:64] FLAG: --tls-private-key-file=""
I0705 06:33:08.714140       1 flags.go:64] FLAG: --tls-reload-interval="1m0s"
I0705 06:33:08.714145       1 flags.go:64] FLAG: --upstream="http://127.0.0.1:8080/"
I0705 06:33:08.714149       1 flags.go:64] FLAG: --upstream-ca-file=""
I0705 06:33:08.714152       1 flags.go:64] FLAG: --upstream-client-cert-file=""
I0705 06:33:08.714156       1 flags.go:64] FLAG: --upstream-client-key-file=""
I0705 06:33:08.714159       1 flags.go:64] FLAG: --upstream-force-h2c="false"
I0705 06:33:08.714162       1 flags.go:64] FLAG: --v="10"
I0705 06:33:08.714166       1 flags.go:64] FLAG: --version="false"
I0705 06:33:08.714176       1 flags.go:64] FLAG: --vmodule=""
W0705 06:33:08.714717       1 kube-rbac-proxy.go:155]

==== Deprecation Warning ======================

Insecure listen address will be removed.
Using --insecure-listen-address won't be possible!

The ability to run kube-rbac-proxy without TLS certificates will be removed.

Not using --tls-cert-file and --tls-private-key-file won't be possible!
For more information, please go to https://github.com/brancz/kube-rbac-proxy/issues/187
===============================================
I0705 06:33:08.714742       1 kube-rbac-proxy.go:284] Valid token audiences:
I0705 06:33:08.714802       1 kube-rbac-proxy.go:378] Generating self signed cert as no cert is provided
I0705 06:33:16.712947       1 kube-rbac-proxy.go:442] Starting TCP socket on 0.0.0.0:8443
I0705 06:33:16.713010       1 kube-rbac-proxy.go:490] Starting TCP socket on 0.0.0.0:10443
I0705 06:33:16.714044       1 kube-rbac-proxy.go:449] Listening securely on 0.0.0.0:8443
I0705 06:33:16.714142       1 kube-rbac-proxy.go:497] Listening securely on 0.0.0.0:10443 for proxy endpoints

  1. Logs for [operator-controller-manager] container
  • I’m suspicious of the word “PDB” and “policy/v1”. Since “policy/v1beta1” was deprecated in Kuberentes 1.21 and is no longer served in kubernetes 1.25. The version of My K8s cluster is 1.20.
  • The below logs are looped with a 15 minutes interval
{"level":"debug","ts":"2024-07-18T04:39:39.197Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"004511da-9742-4c3c-848b-e5f118258450","name":"test-opensearch-cluster-master","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"debug","ts":"2024-07-18T04:39:39.199Z","msg":"resource is in sync","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"004511da-9742-4c3c-848b-e5f118258450","name":"test-opensearch-cluster-data","namespace":"test-opensearch-cluster","apiVersion":"v1","kind":"Service"}
{"level":"error","ts":"2024-07-18T04:39:39.234Z","msg":"Reconciler error","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"test-opensearch-cluster","namespace":"test-opensearch-cluster"},"namespace":"test-opensearch-cluster","name":"test-opensearch-cluster","reconcileID":"004511da-9742-4c3c-848b-e5f118258450","error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource; failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorCauses":[{"error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorVerbose":"failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource\ngetting resource failed\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).delete\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:651\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:523\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s.K8sClientImpl.ReconcileResource\n\t/workspace/pkg/reconcilers/k8s/client.go:198\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).handlePDB\n\t/workspace/pkg/reconcilers/cluster.go:410\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).reconcileNodeStatefulSet\n\t/workspace/pkg/reconcilers/cluster.go:240\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).Reconcile\n\t/workspace/pkg/reconcilers/cluster.go:111\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to delete resource"},{"error":"failed to delete resource: getting resource failed: failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource","errorVerbose":"failed to get API group resources: unable to retrieve the complete list of server APIs: policy/v1: the server could not find the requested resource\ngetting resource failed\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).delete\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:651\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/operator-tools@v0.30.0/pkg/reconciler/resource.go:523\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s.K8sClientImpl.ReconcileResource\n\t/workspace/pkg/reconcilers/k8s/client.go:198\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).handlePDB\n\t/workspace/pkg/reconcilers/cluster.go:410\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).reconcileNodeStatefulSet\n\t/workspace/pkg/reconcilers/cluster.go:240\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*ClusterReconciler).Reconcile\n\t/workspace/pkg/reconcilers/cluster.go:111\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:328\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:143\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nfailed to delete resource"}],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
6

@yeonghyeonKo Have you tried deployment using OpenSearchCluster object and kubectl instead of ArgoCD Operator?

7

@yeonghyeonKo You could be correct with that policy/v1 and PDB. PDB is created policy/v1 apiVersion when OpenSearchCluster object is deployed with pdb definition.

Maybe ArgoCD creates pdb by default. You can control it at the OpenSearchCluster manifest level. When I deploy the following example I don’t get pdb created.

8

@pablo
As you said, I tried to deploy CRDs through kubectl(`kustomize build . --enable-helm > temp.yaml), instead of ArgoCD.

However, deploying with temp.yaml (including policy/v1, not policy/v1beta, apiVersion of PDB resource) failed. So I did change the version of Kubernetes Cluster itself, from 1.20.1 to 1.25.6)

(See [BUG] OpenSearch Dashboards is NOT Deployed after the cluster(master 3, data2) configuration finished. · Issue #853 · opensearch-project/opensearch-k8s-operator · GitHub)

1 Like