Struggling with REST API Access on K8s Deployment

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch Helm Chart: 2.21.0
OpenSearch: 2.15.0
nginx Helm Chart: ingnress-nginx-4.10.1
Kubernetes: 1.28.7

Describe the issue:

Access the REST API as super admin from outside the cluster always gives Authentication Finally Failed.

When tested from another opensearch pod I could use the desired certificates to access the REST API as super admin.

Configuration:

Certificates auto generated by cert manager:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: opensearch-tls-cert
  namespace: opensearch
spec:
  secretName: opensearch-selfsigned-tls-cert
  privateKey:
    rotationPolicy: Always
    algorithm: RSA
    encoding: PKCS8
    size: 2048
  dnsNames:
    - opensearch
  usages:
    - digital signature
    - key encipherment
    - server auth
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer
  literalSubject: 'CN=opensearch,OU=****,O=****,L=****,C=****'
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: opensearch-admin-tls-cert
  namespace: opensearch
spec:
  secretName: opensearch-selfsigned-admin-tls-cert
  privateKey:
    rotationPolicy: Always
    algorithm: RSA
    encoding: PKCS8
    size: 2048
  dnsNames:
    - opensearch
  usages:
    - digital signature
    - key encipherment
    - server auth
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer
  literalSubject: 'CN=admin,OU=****,O=****,L=****,C=****'

Secrets Mounted in install-opensearch-helm.yaml

secretMounts:
            - name: opensearch-tls
              secretName: opensearch-selfsigned-tls-cert
              path: usr/share/opensearch/config/tls/opensearch
            - name: opensearch-admin-tls
              secretName: developers-tls-certificate
              path: usr/share/opensearch/config/tls/opensearch/admin

Opensearch Security Config

plugins:
    security:
      ssl_cert_reload_enabled: true
      ssl:
        transport:
          enabled: true
          pemcert_filepath: tls/opensearch/tls.crt
          pemkey_filepath: tls/opensearch/tls.key
          pemtrustedcas_filepath: tls/opensearch/ca.crt
          enforce_hostname_verification: false
          truststore_filepath: /usr/share/opensearch/config/truststore/cacerts.jks
        http:
          enabled: true
          pemcert_filepath: tls/opensearch/admin/tls.crt
          pemkey_filepath: tls/opensearch/admin/tls.key
          pemtrustedcas_filepath: tls/opensearch/admin/ca.crt
          truststore_filepath: /usr/share/opensearch/config/truststore/cacerts.jks
      nodes_dn:
        - 'CN=opensearch,OU=****,O=****,L=****,C=****'
      allow_unsafe_democertificates: false
      allow_default_init_securityindex: true
      authcz:
        admin_dn:
          - 'CN=admin,OU=****,O=****,L=****,C=****'

nginx Ingress Config

I enabled passthrough as the ingress was originally terminating the TLS connection.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: opensearch
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 10m
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - "wildcard dns for cluster"
      secretName: cluster-certificate
  rules:
    - host: "wildcard dns for cluster"
      http:
        paths:
          - path: /opensearch(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: opensearch-cluster-master-headless
                port:
                  number: 9200

Relevant Logs or Screenshots:
Attempting to use:

curl -v -XPUT https://opensearch.dns/opensearch/_plugins/_security/api/ssl/http/reloadcerts

or

curl -v --cacert cacert.cer --cert tls.pem --key tls.key XPUT https://opensearch.dns/opensearch/_plugins/_security/api/ssl/http/reloadcerts

This is following the docs here: Configuring TLS certificates - OpenSearch Documentation

Returns “Authentication finally failed for null from…”

Verbose Output of Curl

$ curl -XPUT https://opensearch.dns/opensearch/_plugins/_security/api/ssl/http/reloadcerts -v
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 130.246.81.235:443...
* Connected to opensearch.dns (ip) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: C:/Git/mingw64/ssl/certs/ca-bundle.crt
*  CApath: none
} [5 bytes data]
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [5219 bytes data]
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: serialNumber=****; jurisdictionC=****; businessCategory=****; C=****; ST=****; O=****; CN=opensearch.dns
*  start date: Jul 31 00:00:00 2024 GMT
*  expire date: Jul 31 23:59:59 2025 GMT
*  subjectAltName: host "opensearch.dns" matched cert's "opensearch.dns"
*  issuer: C=NL; O=GEANT Vereniging; CN=GEANT EV RSA CA 4
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* h2h3 [:method: PUT]
* h2h3 [:path: /opensearch/_plugins/_security/api/ssl/http/reloadcerts]
* h2h3 [:scheme: https]
* h2h3 [:authority: opensearch.dns]
* h2h3 [user-agent: curl/7.87.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x28179bc1c00)
} [5 bytes data]
> PUT /opensearch/_plugins/_security/api/ssl/http/reloadcerts HTTP/2
> Host: opensaerch.dns
> user-agent: curl/7.87.0
> accept: */*
>
{ [5 bytes data]
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
} [5 bytes data]
< HTTP/2 401
< date: Wed, 31 Jul 2024 14:18:12 GMT
< content-type: text/plain; charset=UTF-8
< content-length: 29
< strict-transport-security: max-age=31536000; includeSubDomains
<
{ [29 bytes data]
100    29  100    29    0     0    517      0 --:--:-- --:--:-- --:--:--   547Authentication finally failed
* Connection #0 to host opensearch.dns left intact
2

I can use the API to view other config e.g:

curl -XGET $BASE/_plugins/_security/api/internalusers/admin --user admin:password -v

When I use --user admin:password with the previous curls I get FORBIDDEN instead of failed.

3

Hi @Ziggiyzoo, for the above you need super admin privileges that are granted then using cert authentication with cert matching the below:

it would look something like this:

curl -XPUT -k -i --cert <admin_cert file name> --key <admin_cert_key file name> https://opensearch.dns/opensearch/_plugins/_security/api/ssl/http/reloadcerts

best,
mj

4

Hi @Mantas ,

Thank you for the response, I’ve tried your suggestion by doing the following.

I’ve create a new self signed admin certificate following the instructions here: Generating self-signed certificates - OpenSearch Documentation with the subject:

CN=admin,OU=***,O=***,L=***,C=***

I have updated the admin_dn to contain:

admin_dn:
  - CN=admin,OU=***,O=***,L=***,C=***

However I am still getting the authentication finally failed error using:

curl -XPUT -k -i --cert admin.pem --key  admin-key.pem --cacert admin.csr $BASE/_plugins/_security/api/ssl/http/reloadcerts

or

curl -XPUT -k -i --cert admin.pem --key  admin-key.pem  $BASE/_plugins/_security/api/ssl/http/reloadcerts

  • My O=*** does contain whitespace, could this be the cause? I’ve double checked to make sure that the CN, OU, O, L, S (state) and C are identical, although I notice state is not used in admin_dn?

  • The admin certificates are not on the cluster, only the transport and HTTP TLS Certificates are? Should the admin certificates be on the cluster?

  • If I really can’t get this to work could I give a user superadmin privelages, or the rights to make these changes, or is that granted through certificate authentication only?

5

Hi @Ziggiyzoo,
Yes, super admin users are authenticated through certificates: Defining users and roles - OpenSearch Documentation

Do you mind sharing commands you used to generate your admin certificate?

Best,
mj

6

Hi @Mantas ,

Sorry for the long reply! But I followed the documentation here: Generating self-signed certificates - OpenSearch Documentation