How Ingress(nginx) controller and Nodes are communicated on https protocol?

DevOps Kubernetes
, , ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch 2.17.1 and opensearch-k8s-operator 2.3.0

Describe the issue:

  • When trying to deploy K8s Ingress API resource for OpenSearch’s service (port: 9200), it requires metadata.annotations to have ingress.kubernetes.io/backend-protocol: https if we want to generate self-signed ca and its certificates for HTTP layer.

  • I’ve already made it possible to implement the below flow and K8s Secrets:

User -(1)-> Ingress Controller(Nginx) -(2)-> Cluster 
# (1) : client(for web browser) certificate
# (2) : backend certificate for HTTP layer created by cert-manager

$ kubectl get secret
NAME                                         TYPE                DATA   AGE
admin-credentials-secret                     Opaque              2      29d
test-opensearch-cluster-admin-cert           kubernetes.io/tls   3      29d
test-opensearch-cluster-admin-password       Opaque              2      29d
test-opensearch-cluster-ca                   kubernetes.io/tls   3      29d
test-opensearch-cluster-http-cert            kubernetes.io/tls   3      29d
test-opensearch-cluster-transport-cert       kubernetes.io/tls   3      29d
client-tls                                   kubernetes.io/tls   2      24h
securityconfig-secret                        Opaque              8      29d

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/backend-protocol: https

    ############################################################################################################################
    # ingress.kubernetes.io/proxy-ssl-name: test-opensearch-cluster  # CN for test-opensearch-cluster-http-cert
    # ingress.kubernetes.io/proxy-ssl-secret: test-opensearch-cluster-ca  # A Secret for CA which would verifies test-opensearch-cluster-http-cert
    ############################################################################################################################
  name: test-opensearch-cluster-os
  namespace: test-opensearch-cluster
spec:
  ingressClassName: nginx
  rules:
  - host: test-opensearch-cluster-os.a.b.com
    http:
      paths:
      - backend:
          service:
            name: test-opensearch-cluster
            port:
              number: 9200
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - test-opensearch-cluster-os.a.b.com
    secretName: client-tls
  • I was just wondering whether the ingress controller has self-signed CA certificates (created by cert-manager) and how it makes handshakes between the cluster itself without specifying proxy-ssl-name & proxy-ssl-secret?
    (Does opensearch-k8s-operator automatically achieve it?)
2

Self-answer : Without the adequate settings for proxy-ssl-*, the client(NGINX ingress controller) won’t verify the server’s certificate so it becomes susceptible to MitM attacks.

  • In a one-way TLS handshake, the server presents its public certificate to the client, and the client is responsible for verifying this certificate against its truststore. This verification step is crucial for ensuring the authenticity and integrity of the server.

  • If proxy-ssl-verify is set to off, the client does not perform the verification step where it checks the server’s certificate against its truststore. Here’s what happens as a result:

    1. Certificate Verification Skipped: The client won’t verify the server’s certificate against its list of trusted certificates. This means the client will accept any certificate presented by the server, even if it’s self-signed, expired, or issued by an untrusted certificate authority (CA).
    2. Vulnerable to Man-in-the-Middle (MitM) Attacks: Because the client is not verifying the server’s certificate, it becomes susceptible to MitM attacks. An attacker could potentially intercept the communication and present their own certificate without detection, thus decrypting or intercepting sensitive data being transmitted.
    3. Loss of Authenticity: The authenticity of the server cannot be guaranteed. Clients are not assured they are communicating with the intended server, compromising both security and trust.

  • It’s important to keep proxy-ssl-verify enabled (set to on) to ensure that the client verifies the server’s certificate. I believe this is a fundamental aspect of establishing a secure TLS connection and helps protect against attacks that could compromise confidential data when it comes to managing databases like OpenSearch.