The OpenSearch Dashboards return a 502 error when accessed via the Ingress route

OpenSearch
,
1

I’m new to OpenSearch, and I used the OpenSearch Operator to deploy an OpenSearch cluster in Kubernetes. I created a Traefik ingress route and nginx ingress resource to access the OpenSearch dashboard, but when I try to access the dashboard via the ingress resource, I receive a 502 error. I checked the logs, and I see the same message being printed, but there are no errors or restarts in the dashboard pods. Could you please assist me with this?

Ingress Error
upstream prematurely closed connection while reading response header from upstream, client:

Thank you !!

2

@athen Could you share your OpenSearchCluster manifest?

3

@pablo
Thank you for your response.
Please find the manifest file attached.
At the moment, I have not configured Logstash, and the cluster is running without any log processing or indexing. This is a fresh cluster

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: my-first
  namespace: default
spec:
  security:
    config:
    tls:
      http:
        generate: true
      transport:
        generate: true
        perNode: true
  general:
    httpPort: 9200
    serviceName: my-first
    version: 2.14.0
    drainDataNodes: true
  dashboards:
    tls:
      enable: true
      generate: true
    version: 2.14.0
    enable: true
    replicas: 1
      #basePath: "/logs"
      #service:
      #type: LoadBalancer
    resources:
      requests:
        memory: "1.5Gi"
        cpu: "1"
      limits:
        memory: "2Gi"
        cpu: "1"    
    additionalConfig:
        server.host: "0.0.0.0"                
        opensearch.requestTimeout: "60000"
        opensearch.shardTimeout: "30000"
        server.maxPayloadBytes: "8388608"
  nodePools:
    - component: masters
      replicas: 3
      resources:
        requests:
          memory: "4Gi"
          cpu: "1000m"
        limits:
          memory: "4Gi"
          cpu: "1000m"
      roles:
        - "data"
        - "cluster_manager"
      persistence:
        emptyDir: {}
4

I added below annotations but it not working still getting same error

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: opensearch-dashboards-ingress
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "75"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "60"
    nginx.ingress.kubernetes.io/proxy-buffers: "8 16k"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "32k"
    nginx.ingress.kubernetes.io/large-client-header-buffers: "4 32k"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: nginx
  rules:
    - host: opensearch-dashboard.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-first-dashboards
                port:
                  number: 5601
5

response time to load the logging page

image
image1507×297 30 KB

I have further troubleshot this issue and noticed that my OpenSearch dashboard takes a significant amount of time to respond to my requests(port-forwarding). which may be contributing to the issue. However, since it is not receiving any data, I’m curious why the response time is still slow. It seems that I may have missed some required configurations for the dashboard. Could you please advise on any configurations that might improve the response time?"

> Ingress error is upstream prematurely closed connection while reading response header from upstream, client:

Additionally, I tried enabling the base directory path and rewrite targets, but I am still getting the same error.

cat config/opensearch_dashboards.yml
opensearch.requestTimeout: 60000
opensearch.shardTimeout: 30000
opensearch.ssl.verificationMode: none
server.basePath: /dashboards
server.host: 0.0.0.0
server.maxPayloadBytes: 8388608
server.name: new-dashboards
server.rewriteBasePath: true
server.ssl.certificate: /usr/share/opensearch-dashboards/certs/tls.crt
server.ssl.enabled: true
server.ssl.key: /usr/share/opensearch-dashboards/certs/tls.key


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: opensearch
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-buffers: "16 16k"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "64k"
    nginx.ingress.kubernetes.io/large-client-header-buffers: "8 64k"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/keep-alive: "600"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - opensearch-dashboard.example.local
      secretName: opensearch-tls-secret
  rules:
    - host: opensearch-dashboard.example.local
      http:
        paths:
          - path: /dashboards(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: new-dashboards
                port:
                  number: 5601

Then I logged into one of my Ingress pods and tried to access the endpoint. I noticed that the Ingress was trying to send the request to the endpoint.

nginx@nginx-ingress-6mvw9:/$ curl -kI https://10.244.3.84:5601/
HTTP/1.1 302 Found
location: /app/login?
osd-name: new-dashboards
cache-control: private, no-cache, no-store, must-revalidate
set-cookie: security_authentication=; Max-Age=0; Expires=Thu, 01 Jan 1970 00:00:00 GMT; HttpOnly; Path=/
content-length: 0
Date: Sun, 03 Nov 2024 06:25:45 GMT
Connection: keep-alive
Keep-Alive: timeout=120

nginx@nginx-ingress-6mvw9:/$ curl -kI https://10.244.3.84:5601/app/login
HTTP/1.1 200 OK
set-cookie: security_authentication=; Max-Age=0; Expires=Thu, 01 Jan 1970 00:00:00 GMT; HttpOnly; Path=/
set-cookie: security_authentication=; Max-Age=0; Expires=Thu, 01 Jan 1970 00:00:00 GMT; HttpOnly; Path=/
content-security-policy: script-src 'unsafe-eval' 'self'; worker-src blob: 'self'; style-src 'unsafe-inline' 'self'
osd-name: new-dashboards
content-type: text/html; charset=utf-8
cache-control: private, no-cache, no-store, must-revalidate
content-length: 117378
vary: accept-encoding
Date: Sun, 03 Nov 2024 06:25:48 GMT
Connection: keep-alive
Keep-Alive: timeout=120
6

The tls setting for your dashboards in the manifest has been enabled.

If you use opensearch-k8s-operator, it’s easy to disable it just by specifying like below:

dashboards:
    enable: true
    replicas: 1
    image: [repo url]
    resources:
      requests:
        memory: "2Gi"
        cpu: "500m"
      limits:
        memory: "2Gi"
        cpu: "500m"
    tls:
      enable: false
    opensearchCredentialsSecret:
      name: admin-credentials-secret
    additionalConfig:
      # https://opensearch.org/docs/latest/install-and-configure/install-dashboards/tls/
      opensearch.ssl.verificationMode: none

If not, your ingress should include a secret for tls setting. Make sure you have the secret in your k8s namespace.

7

Hi @yeonghyeonKo

Thank you for your response. I have enabled TLS settings for my ingress and added a self-signed certificate for it, which is attached to the ingress resource. However, I’ve tried the solution you provided, and unfortunately, I’m still encountering the same error.

Thank you !!

8

@athen Just to clarify, are you using Traefik or Nginx as ingress controller?

9

I’m using nginx ingress controller

10

I was able to resolve this issue after installing kubernetes/nginx-ingress.

https://kubernetes.github.io/ingress-nginx/deploy/

11

@athen I got this working with nginx ingress yesterday but traefik kept giving me 502 when OpenSearch Dashboards was set to HTTPS ( dashboards.tls.enable: true)

12

Hi ,
Initially, I tried using the Traefik Ingress Controller, but I encountered the same error. The controller logs did not provide much information. Then, I switched to NGINX, and I had previously installed the F5 NGINX Ingress Controller, which also triggered the 502 error. However, the Kubernetes Ingress worked fine.

1 Like