404 Error when attempting to connect to running Dashboard Pod

OpenSearch Dashboards
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch: 2.8.0
OpenSearchDashboards: 2.8.0

Describe the issue:
I am attempting to deploy a scaling cluster using OpenSearch Operator & OpenSearch Cluster Helm charts.

I have managed to get the deployment of both working, but when attempting to access the OpenSearch Dashboard I get a 404 Error.

K8s Ingress & Service: Nginx ingress & following service seem to be working fine, certificates are all good & valid.

Pod Logs: When I try to access the dashboard the pod logs a get404 response for my request, so traffic is getting to the pod.

Configuration:

Ansible file managing the Helm install:

- name: Install required Helm packages
  hosts: localhost
  gather_facts: false
  connection: local
  tasks:
    - name: Add Opensearch Operator Helm repo
      kubernetes.core.helm_repository:
        binary_path: ~/bin/helm
        repo_name: opensearch-operator
        repo_url: https://opensearch-project.github.io/opensearch-k8s-operator/
        state: present

    - name: Install Opensearch Operator
      kubernetes.core.helm:
        binary_path: ~/bin/helm
        release_name: opensearch-cluster
        release_namespace: opensearch
        chart_ref: opensearch-operator/opensearch-cluster
        update_repo_cache: yes
        wait: yes
        # For referance this is just the contents of a values.yml
        # https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/charts/opensearch-cluster/values.yaml
        values:
          opensearchCluster:
            general:
              serviceName: "opensearch-cluster"
              httpPort: 9200
              version: 2.8.0
              setVMMaxMapCount: true
            dashboards:
              enable: true
              version: 2.8.0
              replicas: 1
              basePath: "/opensearch-dashboards"
            nodePools:
              - component: masters
                replicas: 2
                roles: # The roles the nodes should have
                  - "cluster_manager"
                  - "data"
                persistence:
                  pvc:
                    storageClass: opensearch
                    accessModes:
                    - ReadWriteMany
              - component: nodes
                replicas: 2
                roles:
                  - "data"
            security:
              config:
                securityConfigSecret:
                  name: opensearch-security-config
              tls:  # Everything related to TLS configuration. OpenSearch will not automatically renew these certifiactes.
                transport:  # Configuration of the transport endpoint
                  generate: true  # Have the operator generate and sign certificates
                  perNode: true  # Separate certificate per node
                http:  # Configuration of the HTTP endpoint
                  generate: true  # Have the Operator generate and sign certificates

Relevant Logs or Screenshots:

2

@Ziggiyzoo Did you resolve your issue?
Could you share opensearch_dashboards.yml from the running pod?
Would you mind sharing the ingress definition?
Have you noticed errors in the ingress controller?

3

No not yet. I went back to using a static deployment instead while I work out a few issues I was having with that!

4

@Ziggiyzoo I’ve tested your Ansible playbook in my k8s cluster and I had no issues.
This is my ingress.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
  name: osd1
  namespace: opensearch
spec:
  ingressClassName: nginx
  rules:
  - host: kube-1.pablo.local
    http:
      paths:
      - backend:
          service:
            name: opensearch-cluster-dashboards
            port:
              number: 5601
        path: /opensearch-dashboards
        pathType: Prefix
  tls:
  - hosts:
     - kube-1.pablo.local
    secretName: proxycert

image
image791Ă—372 13.6 KB

Did you test with kubectl port-forward?
i.e.

kubectl port-forward -n opensearch svc/opensearch-cluster-dashboards 5601:5601 --address 0.0.0.0
5

@pablo I will try port forwarding. I didn’t think this would be an issue as it’s on a different ingress, service & namespace, but I have a static deployment of OpenSearch Dashboard deployed & running.

The ingress it uses the same hostname but the path is different:

Static Dashboard: /opensearch-dashboards
Cluster Dashboard: /opensearch-cluster-dashboards

Update:

Changing the cluster path to be /opensearch-dashboards had no effect. Port forwarding also continues to give the same error.

404 for me on my brower:

{
  "statusCode": 404,
  "error": "Not Found",
  "message": "Not Found"
}

and there is a log showing up on the dashboard pod saying that it got my request and returned 404.

6

What do you mean by that? Are you running two different OpenSearch Dashboards?

7

Yea, I’ve got a OpenSearch deployment that uses the Dashboard installed with Helm from here OpenSearch Project · GitHub

I also have the OpenSearch dashboard that’s deployed when using the Operator from that Helm chart…

I have tried with both running, and running each separately.

8

@Ziggiyzoo I understand that both OpenSearch Dashboards have own basePath in opensearch_dashboards.yml
Also, I assume that each OpenSearch Dashboards has separate service and that is reflected in the ingress. In that case you should have to entries for each OpenSearch Dashboards service.

Additionally, please check the endpoints in the service. Each service should contain IP of the corresponding OpenSearch Dashboards pod.

9

I got it working eventually! Uninstalling it and moving to a different namespace helped in the end as I was confusing myself with conflicting deployments… Not sure what the error was in the end though!

Solution: Keep namespaces separate & double, triple check all your configs :smiley:

1 Like