Unable to create opensearch cluster on kubernetes with opensearchoperator

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

opensearch: 2.4.1
operator: 2.2 opensearch-operator 2.2.0 · opster/opensearch-operator

Describe the issue:

We are trying to explore opensearch, while trying to create opensearch cluster on our cloud using kubernetes we are getting below errors, and the deployment gets stuck.

[2023-01-12T15:10:43,411][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-poc-masters-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)

I am trying to bring the cluster from opensearch-k8s-operator/opensearch-operator/examples at main · Opster/opensearch-k8s-operator · GitHub

Configuration:
Below is the config file I am using.

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: opensearch-poc
  namespace: default
spec:
  security:
    config:
    tls:
       http:
         generate: true
       transport:
         generate: true
         perNode: true
  general:
    #version: 1.3.0
    version: 2.4.1
    httpPort: 9200
    vendor: opensearch
    serviceName: opensearch-poc
    pluginsList: ["repository-s3"]
      #pluginsList: ["repository-s3"," https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/1.3.0.0/prometheus-exporter-1.3.0.0.zip"]
  dashboards:
    tls:
      enable: true
      generate: true
    version: 2.4.1
    enable: true
    replicas: 1
    resources:
      requests:
         memory: "1Gi"
         cpu: "500m"
      limits:
         memory: "2Gi"
         cpu: "500m"
  confMgmt:
    smartScaler: true
  nodePools:
    - component: masters
      replicas: 3
      diskSize: "30Gi"
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "4Gi"
            cpu: "500m"
      roles:
        - "master"
        - "data"
    - component: nodes
      replicas: 3
      diskSize: "100Gi"
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "4Gi"
            cpu: "500m"
      roles:
        - "data"
    - component: coordinators
      replicas: 1
      diskSize: "30Gi"
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "4Gi"
            cpu: "500m"
      roles:
        - "ingest"

Relevant Logs or Screenshots:
$kubectl get pods
NAME READY STATUS RESTARTS AGE
external-dns-559c45cdc-rkbww 2/2 Running 0 3h32m
opensearch-operator-controller-manager-78d85945c6-pm422 3/3 Running 1 (7h55m ago) 7h55m
opensearch-poc-bootstrap-0 2/2 Running 0 21m
opensearch-poc-coordinators-0 1/2 Running 0 21m
opensearch-poc-dashboards-5bbbdb6db7-wp842 1/2 Running 6 (43s ago) 21m
opensearch-poc-masters-0 1/2 Running 0 21m
opensearch-poc-nodes-0 1/2 Running 5 (80s ago) 21m
opensearch-poc-securityconfig-update-55s7x 2/2 Running 0 21m

Could some one please help me understand the issue and point me on how to fix it.

2

Hey @narendrareddy.ala, I see for masters component the role was added as master should be cluster_manager opensearch-k8s-operator/opensearch-v2-cluster.yaml at main · Opster/opensearch-k8s-operator · GitHub, can you try updating that field, remove the existing PVC’s (since you are testing, if not please backup your data) and try to re-deploy your yaml file.
Thank you

2 Likes
3

Thanks for the quick update, we figured the issue is with security plugin being active while istio is also installed on the cluster, so there is a conflict. Is there a way to disable security plugin on open search so that istio can takes care of it?

4

Hey Naren the operator ensures the OpenSearch is started with security and does not bypass this for security compliance as this makes the operator unsafe to use opensearch without security. Can you check if istio can bypass the https check? Also one solution is have a middle haproxy or nginx that can run with http frontend and does the ssl termination sending the traffic to https opensearch with security installed.

5

Hi Prudhvi,
Thanks for the update, I ended up using nginx ingress instead of istio as we were not able to bypass that check.

Thanks and Regards,
Narendra.

1 Like