# OpenSearch Dashboard fails with 502 Bad gateway with new setup

**URL:** <https://forum.opensearch.org/t/opensearch-dashboard-fails-with-502-bad-gateway-with-new-setup/18536>\
**Category:** OpenSearch Dashboards\
**Tags:** install\
**Created:** [March 25, 2024, 9:16am UTC](https://forum.opensearch.org/t/opensearch-dashboard-fails-with-502-bad-gateway-with-new-setup/18536 "2024-03-25T09:16:04Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ar.shashikumar](https://avatars.discourse-cdn.com/v4/letter/a/6a8cbe/32.png) [@ar.shashikumar](https://forum.opensearch.org/u/ar.shashikumar)\
**Post date:** [March 25, 2024, 9:16am UTC](https://forum.opensearch.org/t/opensearch-dashboard-fails-with-502-bad-gateway-with-new-setup/18536/1 "2024-03-25T09:16:04Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):  
OS: Ubuntu 22.04  
k8s version: 1.22.17  
Opensource version:  
opensearch-2.18.0  
opensearch-dashboards-2.16.0

**Describe the issue** :  
Charts used to setup [helm-charts/charts at main · opensearch-project/helm-charts · GitHub](https://github.com/opensearch-project/helm-charts/tree/main/charts)

1. Setting up Opensearch for the first time, Opensearch pods are up and running and able to curl within the pod with 9200
2. Setup of Opensearch-dashboards with pods are in up and running state.

Getting 502 Bad gateway error while browsing the URL  
a. updated ingress to point to the right URL under values.yaml  
b. created ingress for opensearch-dashboards

Observed that if I do not create ingress, it throws an error 404 Not found. Once ingress is created, it started with 502 Bad gateway.

**Configuration** :  
values.yaml of opensearch-dashboards are attached here with

**Relevant Logs or Screenshots** :

**ingress.yaml**

```auto
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
      nginx.ingress.kubernetes.io/secure-backends: "true"
      nginx.io/backend-protocol: HTTPS
      nginx.io/secure-backends: "true"
    creationTimestamp: "2024-03-25T08:44:55Z"
    generation: 1
    name: opensearch-dashboard-ingress
    namespace: os
    resourceVersion: "129757396"
    uid: 8feb1954-85e8-4440-8c93-c50ba927ee2f
  spec:
    rules:
    - host: explore.abc.xyz.com
      http:
        paths:
        - backend:
            service:
              name: os-dashboard-opensearch-dashboards
              port:
                number: 5601
          path: /
          pathType: ImplementationSpecific
    tls:
    - hosts:
      - explore.abc.xyz.com
      secretName: explore.abc.xyz.com

```

**Values.yaml**

```auto
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0

# Default values for opensearch-dashboards.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

opensearchHosts: "https://opensearch-cluster-master:9200"
replicaCount: 1

image:
  repository: "opensearchproject/opensearch-dashboards"
  # override image tag, which is .Chart.AppVersion by default
  tag: ""
  pullPolicy: "IfNotPresent"

startupProbe:
  tcpSocket:
    port: 5601
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 20
  successThreshold: 1
  initialDelaySeconds: 10

livenessProbe:
  tcpSocket:
    port: 5601
  periodSeconds: 20
  timeoutSeconds: 5
  failureThreshold: 10
  successThreshold: 1
  initialDelaySeconds: 10

readinessProbe:
  tcpSocket:
    port: 5601
  periodSeconds: 20
  timeoutSeconds: 5
  failureThreshold: 10
  successThreshold: 1
  initialDelaySeconds: 10

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ""

rbac:
  create: true

# A list of secrets and their paths to mount inside the pod
# This is useful for mounting certificates for security and for mounting
# the X-Pack license
secretMounts: []
# - name: certs
# secretName: dashboard-certs
# path: /usr/share/dashboards/certs

podAnnotations: {}

# Deployment annotations
dashboardAnnotations: {}

extraEnvs: []
# - name: "NODE_OPTIONS"
# value: "--max-old-space-size=1800"

envFrom: []

extraVolumes: []
  # - name: extras
  # emptyDir: {}

extraVolumeMounts: []
  # - name: extras
  # mountPath: /usr/share/extras
  # readOnly: true

extraInitContainers: ""

extraContainers: ""

podSecurityContext: {}

securityContext:
  capabilities:
    drop:
      - ALL
  # readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000

config: {}
  # Default OpenSearch Dashboards configuration from docker image of Dashboards

  # opensearch_dashboards.yml: |
  # server:
  # name: dashboards
  # host: "{{ .Values.serverHost }}"

  # opensearch_dashboards.yml:
  # server:
  # name: dashboards
  # host: "{{ .Values.serverHost }}"

  # Dashboards TLS Config (Ensure the cert files are present before enabling SSL
  # ssl:
  # enabled: true
  # key: /usr/share/opensearch-dashboards/certs/dashboards-key.pem
  # certificate: /usr/share/opensearch-dashboards/certs/dashboards-crt.pem

  # determines how dashboards will verify certificates (needs to be none for default opensearch certificates to work)
  # opensearch:
  # ssl:
  # certificateAuthorities: /usr/share/opensearch-dashboards/certs/dashboards-root-ca.pem
  # if utilizing custom CA certs for connection to opensearch, provide the CA here

opensearchDashboardsYml:
  defaultMode:
  # value should be 0-0777

priorityClassName: ""

opensearchAccount:
  secret: ""
  keyPassphrase:
    enabled: false

labels: {}

hostAliases: []
# - ip: "127.0.0.1"
# hostnames:
# - "foo.local"
# - "bar.local"

serverHost: "0.0.0.0"

service:
  type: ClusterIP
  # The IP family and IP families options are to set the behaviour in a dual-stack environment
  # Omitting these values will let the service fall back to whatever the CNI dictates the defaults
  # should be
  #
  # ipFamilyPolicy: SingleStack
  # ipFamilies:
  # - IPv4
  port: 5601
  loadBalancerIP: ""
  nodePort: ""
  labels: {}
  annotations: {}
  loadBalancerSourceRanges: []
  # 0.0.0.0/0
  httpPortName: http

ingress:
  enabled: true
  # For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
  # See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
  # ingressClassName: nginx
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
      #nginx.ingress.kubernetes.io/backend-protocol: HTTPS
      # nginx.ingress.kubernetes.io/secure-backends: "true"
      # nginx.io/backend-protocol: HTTPS
      # nginx.io/secure-backends: "true"
  labels: {}
  ingressClassName: nginx
  hosts:
    - host: explore.abc.xyz.com
      paths:
        - path: /
          serviceName: "os-dashboard-opensearch-dashboards"
          #pathType: ImplementationSpecific
          servicePort: 5601
  tls:
    - secretName: explore.abc.xyz.com
      hosts:
        - explore.abc.xyz.com
  # - secretName: chart-example-tls
  # hosts:
  # - chart-example.local

resources:
  requests:
    cpu: "100m"
    memory: "512M"
  limits:
    cpu: "100m"
    memory: "512M"

autoscaling:
  # This requires metrics server to be installed, to install use kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
  # See https://github.com/kubernetes-sigs/metrics-server
  enabled: false
  minReplicas: 1
  maxReplicas: 10
  targetCPUUtilizationPercentage: 80

updateStrategy:
  type: "Recreate"

nodeSelector: {}

tolerations: []

affinity: {}

# This is the pod topology spread constraints
# https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/
topologySpreadConstraints: []

# -- Array of extra K8s manifests to deploy
extraObjects: []
  # - apiVersion: secrets-store.csi.x-k8s.io/v1
  # kind: SecretProviderClass
  # metadata:
  # name: argocd-secrets-store
  # spec:
  # provider: aws
  # parameters:
  # objects: |
  # - objectName: "argocd"
  # objectType: "secretsmanager"
  # jmesPath:
  # - path: "client_id"
  # objectAlias: "client_id"
  # - path: "client_secret"
  # objectAlias: "client_secret"
  # secretObjects:
  # - data:
  # - key: client_id
  # objectName: client_id
  # - key: client_secret
  # objectName: client_secret
  # secretName: argocd-secrets-store
  # type: Opaque
  # labels:
  # app.kubernetes.io/part-of: argocd
  # - |
  # apiVersion: policy/v1
  # kind: PodDisruptionBudget
  # metadata:
  # name: {{ template "opensearch-dashboards.fullname" . }}
  # labels:
  # {{- include "opensearch-dashboards.labels" . | nindent 4 }}
  # spec:
  # minAvailable: 1
  # selector:
  # matchLabels:
# {{- include "opensearch-dashboards.selectorLabels" . | nindent 6 }}

# pod lifecycle policies as outlined here:
# https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks
lifecycle: {}
  # preStop:
  # exec:
  # command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
  # postStart:
  # exec:
  # command:
  # - bash
  # - -c
  # - |
  # #!/bin/bash
  # curl -I "http://admin:admin@127.0.0.1:5601/status -H "kbn-xsrf: true" -H 'kbn-xsrf: true' -H "Content-Type: application/json"

## Enable to add 3rd Party / Custom plugins not offered in the default OpenSearchDashboards image.
plugins:
  enabled: false
  installList: []
  # - example-fake-plugin-downloadable-url

```

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [March 25, 2024, 9:16pm UTC](https://forum.opensearch.org/t/opensearch-dashboard-fails-with-502-bad-gateway-with-new-setup/18536/2 "2024-03-25T21:16:07Z")

</div>

@ar.shashikumar The default values.yml configures OpenSearch Dashboards as HTTP.  
You can confirm that by checking the logs of the OpenSearch Dashboards pod.

```auto
"Server running at http://0.0.0.0:5601"

```

Your external ingress is configured to communicate in the backend (ingress - OpenSearch Dashboards) with HTTPS. As a result, you have an HTTPS to HTTP connection which will fail (Bad Gateway).

You don’t need that extra ingress as the one in values.yml is enough. Also, only ingressClassName should be used with this version of Kubernetes. Try the following configuration.

```auto
ingress:
  enabled: true
  # For Kubernetes >= 1.18 you should specify the ingress-controller via the field ingressClassName
  # See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress
  ingressClassName: nginx
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  labels: {}
  hosts:
    - host: explore.abc.xyz.com
      paths:
        - path: /
          backend:
            serviceName: "os-dashboard-opensearch-dashboards"
            servicePort: 5601
  tls:
    - secretName: explore.abc.xyz.com
      hosts:
        - explore.abc.xyz.com

```

Also, remember to clean up any ingresses related to OpenSearch Dashboards before running the helm charts.

---

<div class="post-metadata">

**Author:** ![ar.shashikumar](https://avatars.discourse-cdn.com/v4/letter/a/6a8cbe/32.png) [@ar.shashikumar](https://forum.opensearch.org/u/ar.shashikumar)\
**Post date:** [March 26, 2024, 3:27am UTC](https://forum.opensearch.org/t/opensearch-dashboard-fails-with-502-bad-gateway-with-new-setup/18536/3 "2024-03-26T03:27:49Z")

</div>

@pablo Thanks for the clarification on HTTP vs HTTPS.

**Steps followed post your suggestion:**

1. Deleted externally created ingress  
kos get ingress  
No resources found in os namespace

2. Modified values.yml as per your suggestions and uncommented ```  
ingressClassName: nginx

3. Re-installed opensearch-dashboard

**Result/Findings:**  
When I browse the URL [https://explore.abc.xyz.com](https://explore.abc.xyz.com), I am getting 404 Not Found Nginx error.

Attached opensource dashboard pod logs for reference

 ![opensearch-dashboard-pod-logs](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/5/5fba65ff1701a10e390a97f549abb3f5b0c5b10e.png)

 ![404](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/2/26a9dd44177ed5d9c53da5416b42a6f4d57d766a.png)

NAME READY STATUS RESTARTS AGE  
opensearch-cluster-master-0 1/1 Running 0 23h  
opensearch-cluster-master-1 1/1 Running 0 23h  
opensearch-cluster-master-2 1/1 Running 0 23h  
os-dashboard-opensearch-dashboards-bdb557795-v6x6w 1/1 Running 0 9m

**opensearch pod logs**

[2024-03-26T03:10:52,970][INFO][o.o.a.t.CronTransportAction] [opensearch-cluster-master-0] Start running AD hourly cron.  
[2024-03-26T03:10:52,975][INFO][o.o.a.t.ADTaskManager] [opensearch-cluster-master-0] Start to maintain running historical tasks  
[2024-03-26T03:11:14,057][INFO][o.o.j.s.JobSweeper] [opensearch-cluster-master-0] Running full sweep  
[2024-03-26T03:16:14,058][INFO][o.o.j.s.JobSweeper] [opensearch-cluster-master-0] Running full sweep  
[2024-03-26T03:21:14,061][INFO][o.o.j.s.JobSweeper] [opensearch-cluster-master-0] Running full sweep  
[2024-03-26T03:26:14,062][INFO][o.o.j.s.JobSweeper] [opensearch-cluster-master-0] Running full sweep

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [March 26, 2024, 8:57pm UTC](https://forum.opensearch.org/t/opensearch-dashboard-fails-with-502-bad-gateway-with-new-setup/18536/4 "2024-03-26T20:57:06Z")

</div>

@ar.shashikumar Did you deploy the ingress controller (not the ingress)? Did you check its logs?

Could you share the output of the following command?

```auto
kubectl get pod -A

```

---

<div class="post-metadata">

**Author:** ![ar.shashikumar](https://avatars.discourse-cdn.com/v4/letter/a/6a8cbe/32.png) [@ar.shashikumar](https://forum.opensearch.org/u/ar.shashikumar)\
**Post date:** [March 27, 2024, 2:40am UTC](https://forum.opensearch.org/t/opensearch-dashboard-fails-with-502-bad-gateway-with-new-setup/18536/5 "2024-03-27T02:40:58Z")

</div>

@pablo Nginx ingress controller is running fine in the cluster, however adding ingress component under opensearch-dashboards in values.yaml doesnt seems be creating the ingress.

When we created ingress manually pointing to the opensearch-dashboard service it works fine.

Ideally i prefer using your helm charts to create ingress automatically, but for now we are able to proceed with manual creation of ingress. Let me know if you have any thoughts.

Thanks for all your help @pablo Appreciate you taking the time to help here!!

---

<div class="post-metadata">

**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\
**Post date:** [March 27, 2024, 4:56pm UTC](https://forum.opensearch.org/t/opensearch-dashboard-fails-with-502-bad-gateway-with-new-setup/18536/6 "2024-03-27T16:56:36Z")

</div>

@ar.shashikumar I had no issues creating ingress with official OpenSearch Dashboards helm charts.

Did you delete any ingresses in the cluster that would be mapped to the same OpenSearch Dashboards service and the path before installing helm charts?  
You can’t have two ingresses referring to the same path in the same Kubernetes service.

Also, I’ve noticed that your ingress.yml doesn’t contain ingressClassName. If you used my example then the ingressClassName should be the same as the name of the ingressClass object.

```auto
$ kubectl get ingressclass
NAME CONTROLLER PARAMETERS AGE
nginx k8s.io/ingress-nginx <none> 44h

```

Did you check if your user can create all objects in namespace `os`? Did you use the same user to create external ingress and install helm charts?