Versions
Detected OpenSearch Version: x-content-2.9.0
Detected OpenSearch Security Version: 2.9.0.0
Issue:
I am trying to configure OIDC authentication by using the Open ID Connect endpoint provided by EKS. I hope to use the JWT token that is created for a Kubernetes service account to authenticate to OpenSearch. This means I would not need to maintain passwords in our services.
I have created an entry for the user that matches the “sub” of the JWT token.
Configuration:
config.yml: |-
_meta:
type: "config"
config_version: 2
config:
dynamic:
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
authc:
basic_internal_auth_domain:
description: "Authenticate via HTTP Basic against internal users database"
http_enabled: true
transport_enabled: true
order: 0
http_authenticator:
type: basic
challenge: true
authentication_backend:
type: intern
openid_auth_domain:
http_enabled: true
transport_enabled: true
order: 1
http_authenticator:
type: openid
challenge: false
config:
subject_key: preferred_username
roles_key: roles
openid_connect_url: https://oidc.eks.us-east-2.amazonaws.com/id/XYZZY/.well-known/openid-configuration
verify_hostnames: false
authentication_backend:
type: noop
internal_users.yml: |-
_meta:
type: "internalusers"
config_version: 2
admin:
hash: "$2y$12$eRDrPGUTb95FgBt/PlN5QODnEsbTxU2f2cVp5jEFxuKomLFohKsW6"
reserved: true
backend_roles:
- "admin"
description: "Admin user"
"system:serviceaccount:default:open-search-poc-updater":
hash: "$2y$12$eRDrPGUTb95FgBt/PlN5QODnEsbTxU2f2cVp5jEFxuKomLFohKsW6"
reserved: false
backend_roles:
- "admin"
description: "open-search-poc-updater service account"
The decoded JWT token looks like this:
{
"aud": [
"https://kubernetes.default.svc"
],
"exp": 1724264026,
"iat": 1692728026,
"iss": "https://oidc.eks.us-east-2.amazonaws.com/id/XYZZY",
"kubernetes.io": {
"namespace": "default",
"pod": {
"name": "test",
"uid": "XYZZY"
},
"serviceaccount": {
"name": "open-search-poc-updater",
"uid": "XYZZY"
},
"warnafter": 1692731633
},
"nbf": 1692728026,
"sub": "system:serviceaccount:default:open-search-poc-updater"
}
I am able to get the JWT token and include it as a Bearer token, but I get an Unauthorized result.
Should this work? Am I missing something in configuration
Relevant Logs or Screenshots:
curl --no-progress-meter -v -k -H "Authorization: Bearer $token" "https://open-search-poc.default.svc.cluster.local:9200/?pretty"
* processing: https://open-search-poc.default.svc.cluster.local:9200/?pretty
* Trying 172.20.120.244:9200...
* Connected to open-search-poc.default.svc.cluster.local (172.20.120.244) port 9200
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
* subject: DC=de; L=test; O=node; OU=node; CN=node-0.example.com
* start date: Apr 22 03:43:47 2018 GMT
* expire date: Apr 19 03:43:47 2028 GMT
* issuer: DC=com; DC=example; O=Example Com Inc.; OU=Example Com Inc. Root CA; CN=Example Com Inc. Root CA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.x
> GET /?pretty HTTP/1.1
> Host: open-search-poc.default.svc.cluster.local:9200
> User-Agent: curl/8.2.1
> Accept: */*
> Authorization: Bearer eyJhb<redacted>
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Basic realm="OpenSearch Security"
< content-type: text/plain; charset=UTF-8
< content-length: 12
<
* Connection #0 to host open-search-poc.default.svc.cluster.local left intact
Unauthorizedtest:/#
I see this in the OpenSearch logs:
[2023-08-22T19:09:32,662][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-master-2] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2023-08-22T19:09:32,736][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-master-1] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2023-08-22T19:09:32,740][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-master-1] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2023-08-22T19:09:32,741][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-master-1] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2023-08-22T19:09:32,749][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-master-2] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2023-08-22T19:09:32,785][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-master-0] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2023-08-22T19:09:32,824][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-master-0] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2023-08-22T19:09:32,825][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-master-0] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2023-08-22T19:09:36,789][WARN ][o.o.s.h.HTTPBasicAuthenticator] [opensearch-cluster-master-1] No 'Basic Authorization' header, send 401 and 'WWW-Authenticate Basic'
[2023-08-22T19:09:36,791][WARN ][o.o.s.a.BackendRegistry ] [opensearch-cluster-master-1] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'
I think the “No ‘Authorization’ header” warnings are from communications between nodes in the cluster. The “No ‘Basic Authorization’ header” seems to be from my attempt to curl to the cluster.