OpenSearch Dashboard fails with 502 Bad gateway with new setup

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OS: Ubuntu 22.04
k8s version: 1.22.17
Opensource version:
opensearch-2.18.0
opensearch-dashboards-2.16.0

Describe the issue:
Charts used to setup helm-charts/charts at main · opensearch-project/helm-charts · GitHub

  1. Setting up Opensearch for the first time, Opensearch pods are up and running and able to curl within the pod with 9200
  2. Setup of Opensearch-dashboards with pods are in up and running state.

Getting 502 Bad gateway error while browsing the URL
a. updated ingress to point to the right URL under values.yaml
b. created ingress for opensearch-dashboards

Observed that if I do not create ingress, it throws an error 404 Not found. Once ingress is created, it started with 502 Bad gateway.

Configuration:
values.yaml of opensearch-dashboards are attached here with

Relevant Logs or Screenshots:

ingress.yaml

apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
      nginx.ingress.kubernetes.io/secure-backends: "true"
      nginx.io/backend-protocol: HTTPS
      nginx.io/secure-backends: "true"
    creationTimestamp: "2024-03-25T08:44:55Z"
    generation: 1
    name: opensearch-dashboard-ingress
    namespace: os
    resourceVersion: "129757396"
    uid: 8feb1954-85e8-4440-8c93-c50ba927ee2f
  spec:
    rules:
    - host: explore.abc.xyz.com
      http:
        paths:
        - backend:
            service:
              name: os-dashboard-opensearch-dashboards
              port:
                number: 5601
          path: /
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - explore.abc.xyz.com
      secretName: explore.abc.xyz.com

Values.yaml

# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0

# Default values for opensearch-dashboards.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

opensearchHosts: "https://opensearch-cluster-master:9200"
replicaCount: 1

image:
  repository: "opensearchproject/opensearch-dashboards"
  # override image tag, which is .Chart.AppVersion by default
  tag: ""
  pullPolicy: "IfNotPresent"

startupProbe:
  tcpSocket:
    port: 5601
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 20
  successThreshold: 1
  initialDelaySeconds: 10

livenessProbe:
  tcpSocket:
    port: 5601
  periodSeconds: 20
  timeoutSeconds: 5
  failureThreshold: 10
  successThreshold: 1
  initialDelaySeconds: 10

readinessProbe:
  tcpSocket:
    port: 5601
  periodSeconds: 20
  timeoutSeconds: 5
  failureThreshold: 10
  successThreshold: 1
  initialDelaySeconds: 10

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ""

rbac:
  create: true

# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
secretMounts: []
#  - name: certs
#    secretName: dashboard-certs
#    path: /usr/share/dashboards/certs

podAnnotations: {}

# Deployment annotations
dashboardAnnotations: {}

extraEnvs: []
#  - name: "NODE_OPTIONS"
#    value: "--max-old-space-size=1800"

envFrom: []

extraVolumes: []
  # - name: extras
  #   emptyDir: {}

extraVolumeMounts: []
  # - name: extras
  #   mountPath: /usr/share/extras
  #   readOnly: true

extraInitContainers: ""

extraContainers: ""

podSecurityContext: {}

securityContext:
  capabilities:
    drop:
      - ALL
  # readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000

config: {}
  # Default OpenSearch Dashboards configuration from docker image of Dashboards

  #  opensearch_dashboards.yml: |
  #   server:
  #     name: dashboards
  #     host: "{{ .Values.serverHost }}"

  #  opensearch_dashboards.yml:
  #   server:
  #     name: dashboards
  #     host: "{{ .Values.serverHost }}"


  # Dashboards TLS Config (Ensure the cert files are present before enabling SSL
  # ssl:
  #   enabled: true
  #   key: /usr/share/opensearch-dashboards/certs/dashboards-key.pem
  #   certificate: /usr/share/opensearch-dashboards/certs/dashboards-crt.pem

  # determines how dashboards will verify certificates (needs to be none for default opensearch certificates to work)
  # opensearch:
  #   ssl:
  #     certificateAuthorities: /usr/share/opensearch-dashboards/certs/dashboards-root-ca.pem
  #     if utilizing custom CA certs for connection to opensearch, provide the CA here

opensearchDashboardsYml:
  defaultMode:
  # value should be 0-0777

priorityClassName: ""

opensearchAccount:
  secret: ""
  keyPassphrase:
    enabled: false

labels: {}

hostAliases: []
# - ip: "127.0.0.1"
#   hostnames:
#   - "foo.local"
#   - "bar.local"

serverHost: "0.0.0.0"

service:
  type: ClusterIP
  # The IP family and IP families options are to set the behaviour in a dual-stack environment
  # Omitting these values will let the service fall back to whatever the CNI dictates the defaults
  # should be
  #
  # ipFamilyPolicy: SingleStack
  # ipFamilies:
  # - IPv4
  port: 5601
  loadBalancerIP: ""
  nodePort: ""
  labels: {}
  annotations: {}
  loadBalancerSourceRanges: []
  # 0.0.0.0/0
  httpPortName: http

ingress:
  enabled: true
  # For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
  # See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
  # ingressClassName: nginx
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
      #nginx.ingress.kubernetes.io/backend-protocol: HTTPS
      #    nginx.ingress.kubernetes.io/secure-backends: "true"
      #    nginx.io/backend-protocol: HTTPS
      #    nginx.io/secure-backends: "true"
  labels: {}
  ingressClassName: nginx
  hosts:
    - host: explore.abc.xyz.com
      paths:
        - path: /
          serviceName: "os-dashboard-opensearch-dashboards"
          #pathType: ImplementationSpecific
          servicePort: 5601
  tls:
    - secretName: explore.abc.xyz.com
      hosts:
        - explore.abc.xyz.com
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

resources:
  requests:
    cpu: "100m"
    memory: "512M"
  limits:
    cpu: "100m"
    memory: "512M"

autoscaling:
  # This requires metrics server to be installed, to install use kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
  # See https://github.com/kubernetes-sigs/metrics-server
  enabled: false
  minReplicas: 1
  maxReplicas: 10
  targetCPUUtilizationPercentage: 80

updateStrategy:
  type: "Recreate"

nodeSelector: {}

tolerations: []

affinity: {}

# This is the pod topology spread constraints
# https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/
topologySpreadConstraints: []

# -- Array of extra K8s manifests to deploy
extraObjects: []
  # - apiVersion: secrets-store.csi.x-k8s.io/v1
  #   kind: SecretProviderClass
  #   metadata:
  #     name: argocd-secrets-store
  #   spec:
  #     provider: aws
  #     parameters:
  #       objects: |
  #         - objectName: "argocd"
  #           objectType: "secretsmanager"
  #           jmesPath:
  #               - path: "client_id"
  #                 objectAlias: "client_id"
  #               - path: "client_secret"
  #                 objectAlias: "client_secret"
  #     secretObjects:
  #     - data:
  #       - key: client_id
  #         objectName: client_id
  #       - key: client_secret
  #         objectName: client_secret
  #       secretName: argocd-secrets-store
  #       type: Opaque
  #       labels:
  #         app.kubernetes.io/part-of: argocd
  # - |
  #    apiVersion: policy/v1
  #    kind: PodDisruptionBudget
  #    metadata:
  #      name: {{ template "opensearch-dashboards.fullname" . }}
  #      labels:
  #        {{- include "opensearch-dashboards.labels" . | nindent 4 }}
  #    spec:
  #      minAvailable: 1
  #      selector:
  #        matchLabels:
#          {{- include "opensearch-dashboards.selectorLabels" . | nindent 6 }}

# pod lifecycle policies as outlined here:
# https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks
lifecycle: {}
  # preStop:
  #   exec:
  #     command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
  # postStart:
  #   exec:
  #     command:
  #       - bash
  #       - -c
  #       - |
  #         #!/bin/bash
  #         curl -I "http://admin:admin@127.0.0.1:5601/status -H "kbn-xsrf: true" -H 'kbn-xsrf: true' -H "Content-Type: application/json"

## Enable to add 3rd Party / Custom plugins not offered in the default OpenSearchDashboards image.
plugins:
  enabled: false
  installList: []
  # - example-fake-plugin-downloadable-url

@ar.shashikumar The default values.yml configures OpenSearch Dashboards as HTTP.
You can confirm that by checking the logs of the OpenSearch Dashboards pod.

"Server running at http://0.0.0.0:5601"

Your external ingress is configured to communicate in the backend (ingress - OpenSearch Dashboards) with HTTPS. As a result, you have an HTTPS to HTTP connection which will fail (Bad Gateway).

You don’t need that extra ingress as the one in values.yml is enough. Also, only ingressClassName should be used with this version of Kubernetes. Try the following configuration.

ingress:
  enabled: true
  # For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
  # See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
  ingressClassName: nginx
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  labels: {}
  hosts:
    - host: explore.abc.xyz.com
      paths:
        - path: /
          backend:
            serviceName: "os-dashboard-opensearch-dashboards"
            servicePort: 5601
  tls:
    - secretName: explore.abc.xyz.com
      hosts:
        - explore.abc.xyz.com

Also, remember to clean up any ingresses related to OpenSearch Dashboards before running the helm charts.

@pablo Thanks for the clarification on HTTP vs HTTPS.

Steps followed post your suggestion:

  1. Deleted externally created ingress
    kos get ingress
    No resources found in os namespace

  2. Modified values.yml as per your suggestions and uncommented ```
    ingressClassName: nginx

  3. Re-installed opensearch-dashboard

Result/Findings:
When I browse the URL https://explore.abc.xyz.com, I am getting 404 Not Found Nginx error.

Attached opensource dashboard pod logs for reference

NAME READY STATUS RESTARTS AGE
opensearch-cluster-master-0 1/1 Running 0 23h
opensearch-cluster-master-1 1/1 Running 0 23h
opensearch-cluster-master-2 1/1 Running 0 23h
os-dashboard-opensearch-dashboards-bdb557795-v6x6w 1/1 Running 0 9m

opensearch pod logs

[2024-03-26T03:10:52,970][INFO ][o.o.a.t.CronTransportAction] [opensearch-cluster-master-0] Start running AD hourly cron.
[2024-03-26T03:10:52,975][INFO ][o.o.a.t.ADTaskManager ] [opensearch-cluster-master-0] Start to maintain running historical tasks
[2024-03-26T03:11:14,057][INFO ][o.o.j.s.JobSweeper ] [opensearch-cluster-master-0] Running full sweep
[2024-03-26T03:16:14,058][INFO ][o.o.j.s.JobSweeper ] [opensearch-cluster-master-0] Running full sweep
[2024-03-26T03:21:14,061][INFO ][o.o.j.s.JobSweeper ] [opensearch-cluster-master-0] Running full sweep
[2024-03-26T03:26:14,062][INFO ][o.o.j.s.JobSweeper ] [opensearch-cluster-master-0] Running full sweep

@ar.shashikumar Did you deploy the ingress controller (not the ingress)? Did you check its logs?

Could you share the output of the following command?

kubectl get pod -A

@pablo Nginx ingress controller is running fine in the cluster, however adding ingress component under opensearch-dashboards in values.yaml doesnt seems be creating the ingress.

When we created ingress manually pointing to the opensearch-dashboard service it works fine.

Ideally i prefer using your helm charts to create ingress automatically, but for now we are able to proceed with manual creation of ingress. Let me know if you have any thoughts.

Thanks for all your help @pablo Appreciate you taking the time to help here!!

@ar.shashikumar I had no issues creating ingress with official OpenSearch Dashboards helm charts.

Did you delete any ingresses in the cluster that would be mapped to the same OpenSearch Dashboards service and the path before installing helm charts?
You can’t have two ingresses referring to the same path in the same Kubernetes service.

Also, I’ve noticed that your ingress.yml doesn’t contain ingressClassName. If you used my example then the ingressClassName should be the same as the name of the ingressClass object.

$ kubectl get ingressclass
NAME    CONTROLLER             PARAMETERS   AGE
nginx   k8s.io/ingress-nginx   <none>       44h

Did you check if your user can create all objects in namespace os? Did you use the same user to create external ingress and install helm charts?