Traefik as Ingress defined in Helm chart values

Versions :
Opensearch: 2.19.4
Helm chart: 2.36.0
Traefik: 3.6.8

Describe the issue: Trying to move from nginx to traefik, 3 pods all start, with their services, , ingress and endpoints started / configured correctly(I believe) but getting 502 Bad Gateway. I’m assuming I’m missing a specific annotation OR one too many?. I also see that 2.19.5 has been release days ago, but going through the release notes, nothing specific with regards to traefik support? Any help / guidance would be great.

Configuration: Annotations:
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/service.serversscheme: “https”

Relevant Logs or Screenshots:

@GordonOlson welcome to the forum and thanks for the question.

Would you be able to share your values.yml file for opensearch deployment. I believe it could be reading http and attempting to skip https.

Also, if you are working with self signed certs you will need to skip the validation, using global flag: --serversTransport.insecureSkipVerify=true

I tested this using minikube, but the concept should still work.

Thanks for getting back to me. I am using a known certificated from a valid CA, but did try to update the /usr/share/opensearch/jdk/lib/security/cacert keystore with addditonal certs, with “no” luck, using the keystore: [ ] attribute in the values.yaml with again, no luck, then tried to figure out how to create a new secret for the keystore, but again, no luck, documentation is lacking a bit here. 2 more things, this was a working deployment with nginx, only the ingress section of the values file was updated to traefik. Secondly, here is my redacted values file:

NOTE: annotation section, “.” (dot) replaced with space, as I could not paste as the string was seen as a URL, with a 2 link limit in my post???


clusterName: “opensearch-cluster”
nodeGroup: “master”

singleNode: false

masterService: “opensearch-cluster-master”

roles:

  • master
  • ingest
  • data
  • remote_cluster_client

replicas: 3

majorVersion: “”

global:
dockerRegistry: “”

opensearchHome: /usr/share/opensearch

config:
opensearch.yml: |
cluster.name: opensearch-cluster

network.host: 0.0.0.0

extraEnvs:

  • name: OPENSEARCH_INITIAL_ADMIN_PASSWORD
    value: my_admin_password

envFrom:

secretMounts:

hostAliases:

image:
repository: “my-repo/opensearchproject/opensearch”
tag: “2.19.5”
pullPolicy: “IfNotPresent”

podAnnotations: {}

openSearchAnnotations: {}

labels: {}

opensearchJavaOpts: “-Xmx3500M -Xms3500M”

resources:
requests:
cpu: “3000m”
memory: “4000Mi”

initResources: {}

sidecarResources: {}

networkHost: “0.0.0.0”

rbac:
create: false
serviceAccountAnnotations: {}
serviceAccountName: “”
automountServiceAccountToken: false

podSecurityPolicy:
create: false
name: “”
spec:
privileged: true
fsGroup:
rule: RunAsAny
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:

  • secret
  • configMap
  • persistentVolumeClaim
  • emptyDir

persistence:
enabled: true
enableInitChown: true
image: myrepo/library/busybox
labels:
enabled: false
additionalLabels: {}
accessModes:

  • ReadWriteOnce
    size: 30Gi
    annotations: {}

extraVolumes:

  • name: secrets-store-inline
    csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
    secretProviderClass: “dev-vault”

extraVolumeMounts:

  • name: secrets-store-inline
    mountPath: /mnt/secrets-store
    readOnly: true

extraContainers:

extraInitContainers:

priorityClassName: “”

antiAffinityTopologyKey: “kubernetes io/hostname”

antiAffinity: “soft”

customAntiAffinity: {}

nodeAffinity: {}

podAffinity: {}

topologySpreadConstraints:

podManagementPolicy: “Parallel”

enableServiceLinks: true

protocol: https
httpPort: 9200
transportPort: 9300
metricsPort: 9600
httpHostPort: “”
transportHostPort: “”

service:
labels: {}
labelsHeadless: {}
headless:
annotations: {}
type: ClusterIP
nodePort: “”
annotations: {}
httpPortName: http
transportPortName: transport
metricsPortName: metrics
loadBalancerIP: “”
loadBalancerSourceRanges:
externalTrafficPolicy: “”

updateStrategy: RollingUpdate

maxUnavailable: 1

podSecurityContext:
fsGroup: 1000
runAsUser: 1000

securityContext:
capabilities:
drop:

  • ALL
    runAsNonRoot: true
    runAsUser: 1000

securityConfig:
enabled: true
path: “/usr/share/opensearch/config/opensearch-security”
actionGroupsSecret:
configSecret:
internalUsersSecret:
rolesSecret:
rolesMappingSecret:
tenantsSecret:
config:
securityConfigSecret: “”
dataComplete: true
data: {}

terminationGracePeriod: 120

sysctlVmMaxMapCount: 262144

startupProbe:
tcpSocket:
port: 9200
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 30

livenessProbe: {}

readinessProbe:
tcpSocket:
port: 9200
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 3

schedulerName: “”

imagePullSecrets:
nodeSelector:
app: mylabel
tolerations:

  • effect: NoSchedule
    operator: Equal
    key: app
    value: mylabel

ingress:
enabled: true
ingressClassName: traefik

annotations:
kubernetes io/ingress.class: traefik
traefik ingress kubernetes io/router entrypoints: websecure
traefik ingress kubernetes io/router tls: “true”

numerous configurations not working for me

#traefik ingress kubernetes io/router entrypoints: websecure
#traefik ingress kubernetes io/service serversscheme: “https”
#traefik ingress kubernetes io/service nativelb: “true”
#traefik ingress kubernetes io/router tls: “true”
#traefik ingress kubernetes io/service sticky cookie: “true”
#traefik ingress kubernetes io/service sticky cookie name: op_cookie
#traefik ingress kubernetes io/service sticky cookie secure: “true”
#kubernetes io/tls-acme: “true”
#traefik ingress kubernetes io/router middlewares: traefik-large-file-upload@kubernetescrd
#traefik ingress kubernetes io/service serverstransport: traefik-extended-timeout@kubernetescrd

previous working nginx annotations

#nginx ingress kubernetes io/backend-protocol: “HTTPS”
#nginx ingress kubernetes io/proxy-ssl-verify: “false”
#nginx ingress kubernetes io/proxy-body-size: “0”
ingressLabels: {}
path: /
hosts:

  • opensearch-dev mydomain com
    tls:
  • secretName: mydev-tls
    hosts:
  • opensearch-dev mydomain com

nameOverride: “”
fullnameOverride: “”

masterTerminationFix: false

opensearchLifecycle: {}

lifecycle: {}

keystore:

networkPolicy:
create: false

http:
enabled: false

fsGroup: “”

sysctl:
enabled: false

sysctlInit:
enabled: true
image: myrepo/library/busybox

plugins:
enabled: false
installList:
removeList:

extraObjects:

  • |
    apiVersion: policy/v1
    kind: PodDisruptionBudget
    metadata:
    name: {{ template “opensearch.uname” . }}
    labels:
    {{- include “opensearch.labels” . | nindent 4 }}
    spec:
    minAvailable: 1
    selector:
    matchLabels:
    {{- include “opensearch.selectorLabels” . | nindent 6 }}

serviceMonitor:
enabled: false

path: /_prometheus/metrics

scheme: http

interval: 10s

labels: {}

tlsConfig: {}

basicAuth:
enabled: false

one more thing, I did try and use the ingressLabels: {} attribute but with no luck again, probably a syntax definition for the values which I couldn’t figure out

@GordonOlson in your nginx configuraion you are disabling the cert validation as the backend certificates are still self signed.

The same should be done in traefik configuration. Have you tried using the config I provided in my previous reply: --serversTransport.insecureSkipVerify=true

I have but it isn’t either defined correctly or not be respected:

kubectl describe ingress opensearch-cluster-master
Name: opensearch-cluster-master
Labels: app.kubernetes io/component=opensearch-cluster-master
app.kubernetes io/instance=opensearch-dev
app.kubernetes io/manage-by=Helm
app.kubernetes io/name=opensearch
app.kubernetes io/version=2.19.5
app.kubernetes io/chart=opensearch-2.37.0
seversTransport.insecureSkipVerify=true
Namespace: my-namespace
…..

I’m guessing although it isn’t throwing an exception, it isn’t a valid label?

@GordonOlson I think there is a typo in your config: seversTransport