Hello folks, I’ve found a few topics about cognito but for old versions, so let’s put everything into this topic.
After that, when the solution is found, I’ll create a PR to the docs and will do my best to keep it up to date.
Versions
Versions 2.6.0 from docker hub, running inside EKS (aws k8s), behind AWS ALB Ingress and k8s service.
Describe the issue:
OS Dashboard returns 401 for this endpoints after successful auth with Cognito (OpenID provider):
- GET https://BASE_URL/api/v1/restapiinfo
- GET https://BASE_URL/api/v1/configuration/account
- GET https://BASE_URL/api/v1/auth/type
- GET https://BASE_URL/api/v1/configuration/account
OS logs with confirmation, that auth succeeded:
SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
[2023-04-04T13:59:24,368][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] [opensearch-cluster-master-0] Starting handshake
[2023-04-04T13:59:24,371][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] [opensearch-cluster-master-0] Secure session established
[2023-04-04T13:59:24,371][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] [opensearch-cluster-master-0] negotiated protocol: TLSv1.2
[2023-04-04T13:59:24,371][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] [opensearch-cluster-master-0] negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
[2023-04-04T13:59:24,371][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] [opensearch-cluster-master-0] peer principal: CN=cognito-idp.us-east-1.amazonaws.com
[2023-04-04T13:59:24,371][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] [opensearch-cluster-master-0] peer alternative names: [cognito-idp.us-east-1.amazonaws.com, cognito-idp-fips.us-east-1.amazonaws.com]
[2023-04-04T13:59:24,371][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] [opensearch-cluster-master-0] issuer principal: CN=Amazon RSA 2048 M02, O=Amazon, C=US
[2023-04-04T13:59:24,371][DEBUG][o.a.h.i.c.DefaultHttpClientConnectionOperator] [opensearch-cluster-master-0] Connection established 192.168.2.158:38124<->3.219.186.64:443
[2023-04-04T13:59:24,371][DEBUG][o.a.h.i.c.DefaultManagedHttpClientConnection] [opensearch-cluster-master-0] http-outgoing-1: set socket timeout to 5000
[2023-04-04T13:59:24,371][DEBUG][o.a.h.i.e.MainClientExec ] [opensearch-cluster-master-0] Executing request GET /us-east-pool_id/.well-known/jwks.json HTTP/1.1
[2023-04-04T13:59:24,371][DEBUG][o.a.h.i.e.MainClientExec ] [opensearch-cluster-master-0] Target auth state: UNCHALLENGED
[2023-04-04T13:59:24,371][DEBUG][o.a.h.i.e.MainClientExec ] [opensearch-cluster-master-0] Proxy auth state: UNCHALLENGED
[2023-04-04T13:59:24,371][DEBUG][o.a.h.headers ] [opensearch-cluster-master-0] http-outgoing-1 >> GET /us-east-pool_id/.well-known/jwks.json HTTP/1.1
[2023-04-04T13:59:24,371][DEBUG][o.a.h.headers ] [opensearch-cluster-master-0] http-outgoing-1 >> Host: cognito-idp.us-east-1.amazonaws.com
[2023-04-04T13:59:24,372][DEBUG][o.a.h.headers ] [opensearch-cluster-master-0] http-outgoing-1 >> Connection: Keep-Alive
[2023-04-04T13:59:24,372][DEBUG][o.a.h.headers ] [opensearch-cluster-master-0] http-outgoing-1 >> User-Agent: Apache-HttpClient/4.5.13 (Java/17.0.6)
[2023-04-04T13:59:24,372][DEBUG][o.a.h.headers ] [opensearch-cluster-master-0] http-outgoing-1 >> Accept-Encoding: gzip,deflate
[2023-04-04T13:59:24,372][DEBUG][o.a.h.wire ] [opensearch-cluster-master-0] http-outgoing-1 >> "GET /us-east-pool_id/.well-known/jwks.json HTTP/1.1[\r][\n]"
[2023-04-04T13:59:24,372][DEBUG][o.a.h.wire ] [opensearch-cluster-master-0] http-outgoing-1 >> "Host: cognito-idp.us-east-1.amazonaws.com[\r][\n]"
[2023-04-04T13:59:24,372][DEBUG][o.a.h.wire ] [opensearch-cluster-master-0] http-outgoing-1 >> "Connection: Keep-Alive[\r][\n]"
[2023-04-04T13:59:24,372][DEBUG][o.a.h.wire ] [opensearch-cluster-master-0] http-outgoing-1 >> "User-Agent: Apache-HttpClient/4.5.13 (Java/17.0.6)[\r][\n]"
[2023-04-04T13:59:24,372][DEBUG][o.a.h.wire ] [opensearch-cluster-master-0] http-outgoing-1 >> "Accept-Encoding: gzip,deflate[\r][\n]"
[2023-04-04T13:59:24,372][DEBUG][o.a.h.wire ] [opensearch-cluster-master-0] http-outgoing-1 >> "[\r][\n]"
[2023-04-04T13:59:24,380][DEBUG][o.a.h.wire ] [opensearch-cluster-master-0] http-outgoing-1 << "HTTP/1.1 200 OK[\r][\n]"
[2023-04-04T13:59:24,380][DEBUG][o.a.h.wire ] [opensearch-cluster-master-0] http-outgoing-1 << "Date: Tue, 04 Apr 2023 13:59:24 GMT[\r][\n]"
[2023-04-04T13:59:24,380][DEBUG][o.a.h.wire ] [opensearch-cluster-master-0] http-outgoing-1 << "Content-Type: application/json[\r][\n]"
[2023-04-04T13:59:24,380][DEBUG][o.a.h.wire ] [opensearch-cluster-master-0] http-outgoing-1 << "Content-Length: 916[\r][\n]"
[2023-04-04T13:59:24,380][DEBUG][o.a.h.wire ] [opensearch-cluster-master-0] http-outgoing-1 << "Connection: keep-alive[\r][\n]"
[2023-04-04T13:59:24,380][DEBUG][o.a.h.wire ] [opensearch-cluster-master-0] http-outgoing-1 << "x-amzn-RequestId: e0d57add-d518-4fb7-9a6e-741543096113[\r][\n]"
[2023-04-04T13:59:24,380][DEBUG][o.a.h.wire ] [opensearch-cluster-master-0] http-outgoing-1 << "Cache-Control: public, max-age=86400[\r][\n]"
[2023-04-04T13:59:24,380][DEBUG][o.a.h.wire ] [opensearch-cluster-master-0] http-outgoing-1 << "[\r][\n]"
[2023-04-04T13:59:24,380][DEBUG][o.a.h.wire ] [opensearch-cluster-master-0] http-outgoing-1 << "{"keys":[{"alg":"RS256","e":"AQAB","kid":"a4rf5Qmvj0uWQrUOjamqEq4npT775iWYUr5Ze2RfT90=","kty":"RSA","n":"tuZkoTpJkt0shzCW6z_O
kqrLkNfKiNsQ0R66oZxW8VCmJyPLyyJ_HKvcpoXEAib8jGMTNc4D_mIdumyvOz8PMlPx7Z7ET1Y807jbBF86aTTcF3DjWq_A4DVgTkZpqSmjye89nFX0KGSE0QrwKtoPDhGnxWj1AaNjIvn8qgy3K_tktVAYJeqnTdzaU6FzxQYrCl5xazm7emelok_p_7RJYCDOh4lZKq-csWt-4kGxI7TKaQYZpNFXCYMol7BUb-q
sXOUsnrYkyVeofqi6ZzB2XNIpkJFIN_H1de4vr3dO31wInVMEF0-1upr2ynQwM_vZ7G4idD9rJ5H75LEOTOSW1Q","use":"sig"},{"alg":"RS256","e":"AQAB","kid":"GCLXw2O8cOciedEHoxvhS/IBm1+vemrUtL3HQb4VpF0=","kty":"RSA","n":"ycMCElDdxl2a7t1H05k1y5DboPCF80w9GFWYp
gi1NpFRs-VnwjHixeJGrUQt-SIVjmf1mp_pKlyvnsf3u-m_kfqzmq4SO-fOYULooysrd3WgnTdl_4d4IkibCUYXHoLTIYTJqqzYOf7e2dGrWV-a1nMWn9uKCkvPycyVD-d7ksJ52t4kAcGklX6PCGdRVfghm6ycY4kp-ENh78sj3JeEkxXY2mvhRr9itJpTmcs4dcYiZfvDQbrIxdc3Hzkgo2WUcJRXsZ1Zpd2rYvDl
R4tLj66SDpLoe2teZ9w6HaHomaZI3ea6p8yRAwWQNKlGn-AP86oC-5-v7HtGlQbXCqloxw","use":"sig"}]}"
[2023-04-04T13:59:24,380][DEBUG][o.a.h.headers ] [opensearch-cluster-master-0] http-outgoing-1 << HTTP/1.1 200 OK
[2023-04-04T13:59:24,380][DEBUG][o.a.h.headers ] [opensearch-cluster-master-0] http-outgoing-1 << Date: Tue, 04 Apr 2023 13:59:24 GMT
[2023-04-04T13:59:24,380][DEBUG][o.a.h.headers ] [opensearch-cluster-master-0] http-outgoing-1 << Content-Type: application/json
[2023-04-04T13:59:24,380][DEBUG][o.a.h.headers ] [opensearch-cluster-master-0] http-outgoing-1 << Content-Length: 916
[2023-04-04T13:59:24,380][DEBUG][o.a.h.headers ] [opensearch-cluster-master-0] http-outgoing-1 << Connection: keep-alive
[2023-04-04T13:59:24,380][DEBUG][o.a.h.headers ] [opensearch-cluster-master-0] http-outgoing-1 << x-amzn-RequestId: e0d57add-d518-4fb7-9a6e-741543096113
[2023-04-04T13:59:24,380][DEBUG][o.a.h.headers ] [opensearch-cluster-master-0] http-outgoing-1 << Cache-Control: public, max-age=86400
[2023-04-04T13:59:24,381][DEBUG][o.a.h.i.e.MainClientExec ] [opensearch-cluster-master-0] Connection can be kept alive indefinitely
[2023-04-04T13:59:24,385][DEBUG][o.a.h.i.c.PoolingHttpClientConnectionManager] [opensearch-cluster-master-0] Connection [id: 1][route: {s}->https://cognito-idp.us-east-1.amazonaws.com:443] can be kept alive indefinitely
[2023-04-04T13:59:24,385][DEBUG][o.a.h.i.c.DefaultManagedHttpClientConnection] [opensearch-cluster-master-0] http-outgoing-1: set socket timeout to 0
[2023-04-04T13:59:24,385][DEBUG][o.a.h.i.c.PoolingHttpClientConnectionManager] [opensearch-cluster-master-0] Connection released: [id: 1][route: {s}->https://cognito-idp.us-east-1.amazonaws.com:443][total available: 1; route allocated:
1 of 5; total allocated: 1 of 10]
[2023-04-04T13:59:24,386][DEBUG][o.a.h.i.c.PoolingHttpClientConnectionManager] [opensearch-cluster-master-0] Connection manager is shutting down
[2023-04-04T13:59:24,386][DEBUG][o.a.h.i.c.DefaultManagedHttpClientConnection] [opensearch-cluster-master-0] http-outgoing-1: Close connection
[2023-04-04T13:59:24,386][DEBUG][o.a.h.i.c.PoolingHttpClientConnectionManager] [opensearch-cluster-master-0] Connection manager shut down
[2023-04-04T13:59:24,386][INFO ][c.a.d.a.h.j.k.SelfRefreshingKeySet] [opensearch-cluster-master-0] KeySetProvider finished
[2023-04-04T13:59:24,391][DEBUG][o.o.s.a.BackendRegistry ] [opensearch-cluster-master-0] Rest user 'User [name=punko, backend_roles=[admin, pmm_admins, sonar-administrators], requestedTenant=null]' is authenticated
[2023-04-04T13:59:24,391][DEBUG][o.o.s.a.BackendRegistry ] [opensearch-cluster-master-0] securitytenant 'null'
[2023-04-04T13:59:24,391][TRACE][o.o.s.a.i.AuditLogImpl ] [opensearch-cluster-master-0] Check for REST category:AUTHENTICATED, effectiveUser:punko, request:/_plugins/_security/authinfo
[2023-04-04T13:59:24,391][TRACE][o.o.t.TaskManager ] [opensearch-cluster-master-0] register 704 [transport] [indices:data/write/index] [index {[security-auditlog-2023.04.04][null], source[{"audit_cluster_name":"opensearch-cluster","audit_node_name":"opensearch-cluster-master-0","audit_request_initiating_user":"punko","audit_rest_request_method":"GET","audit_category":"AUTHENTICATED","audit_request_origin":"REST","audit_node_id":"P15x9wCdRoWrE2iUDmEfMw","audit_request_layer":"REST","audit_rest_request_path":"/_plugins/_security/authinfo","@timestamp":"2023-04-04T13:59:24.391+00:00","audit_request_effective_user_is_admin":false,"audit_format_version":4,"audit_request_remote_address":"192.168.2.192","audit_node_host_address":"192.168.2.158","audit_rest_request_headers":{"x-opensearch-product-origin":["opensearch-dashboards"],"Connection":["keep-alive"],"x-opaque-id":["d46b214c-de17-45d2-95a7-6bf0cf16624b"],"Host":["opensearch-cluster-master:9200"],"Content-Length":["0"]},"audit_request_effective_user":"punko","audit_node_host_name":"192.168.2.158"}]}]
So I assume the problem is somewhere in Dashboards side.
Configuration:
/usr/share/opensearch/config/config.yml:
_meta:
type: "config"
config_version: 2
config:
dynamic:
authc:
basic_internal_auth_domain:
http_enabled: true
transport_enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: internal
openid_auth_domain:
http_enabled: true
transport_enabled: true
order: 1
http_authenticator:
type: openid
challenge: false
config:
openid_connect_idp:
enable_ssl: true
pemtrustedcas_filepath: '/usr/share/opensearch/config/cognito-pool.pem' <---chain
verify_hostnames: false
subject_key: 'preferred_username'
roles_key: 'cognito:groups'
openid_connect_url: 'https://cognito-idp.us-east-1.amazonaws.com/us-east-POOL_ID/.well-known/openid-configuration'
authentication_backend:
type: noop
/usr/share/opensearch/config/opensearch.yml
network.host: 0.0.0.0
discovery.type: single-node
plugins:
security:
ssl:
transport:
pemcert_filepath: esnode.pem <---- self-signed, added to pods withing configmap
pemkey_filepath: esnode-key.pem
pemtrustedcas_filepath: root-ca.pem
keystore_type: PKCS12/PFX <--- found somewhere that trustore required despite use of filepath. So have to modify docker's entrypoint and combine all certs into java keystore.
truststore_type: PKCS12/PFX
truststore_filepath: os.pkc
enforce_hostname_verification: false
http:
enabled: true
pemcert_filepath: esnode.pem
keystore_type: PKCS12/PFX
truststore_type: PKCS12/PFX
pemkey_filepath: esnode-key.pem
truststore_filepath: os.pkc
pemtrustedcas_filepath: root-ca.pem
allow_unsafe_democertificates: false
allow_default_init_securityindex: false
authcz:
admin_dn:
- "O=Internet Widgits Pty Ltd,ST=Some-State,C=AU"
audit:
type: internal_opensearch
enable_snapshot_restore_privilege: true
check_snapshot_restore_write_privileges: true
restapi:
roles_enabled: ["all_access", "security_rest_api_access"]
system_indices:
enabled: true
indices:
[
".opendistro-alerting-config",
".opendistro-alerting-alert*",
".opendistro-anomaly-results*",
".opendistro-anomaly-detector*",
".opendistro-anomaly-checkpoints",
".opendistro-anomaly-detection-state",
".opendistro-reports-*",
".opendistro-notifications-*",
".opendistro-notebooks",
".opendistro-asynchronous-search-response*",
]
/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml:
server.host: "https://BASE_URL.com"
opensearch_security.cookie.secure: true
opensearch_security.openid.client_id: qwertyuiop0987654321
opensearch_security.openid.client_secret: 1234567890poiuytrewq
opensearch_security.openid.connect_url: https://cognito-idp.us-east-1.amazonaws.com/us-east-POOL_ID/.well-known/openid-configuration
opensearch_security.openid.scope: "openid profile email"
opensearch.username: admin
opensearch.password: admin_pass
opensearch_security.openid.base_redirect_url: https://BASE_URL
opensearch.requestHeadersAllowlist: ["Authorization","authorization","securitytenant"]
opensearch.ssl.verificationMode: none
opensearch_security.multitenancy.enabled: false
opensearch_security.auth.type: ["basicauth","openid"]
opensearch_security.auth.multiple_auth_enabled: true
opensearch_security.openid.root_ca: /usr/share/opensearch-dashboards/config/cognito-pool.pem <- chain, the same as for config.yml
opensearch_security.openid.verify_hostnames: false
logging.verbose: true
Cognito configuration:
- Authentication flows
ALLOW_REFRESH_TOKEN_AUTH
ALLOW_USER_SRP_AUTH
- Authentication flow session duration
3 minutes
- Refresh token expiration
30 day(s)
- Access token expiration
5 minutes
- ID token expiration
5 minutes
- Allowed callback URLs
https://BASE_URL/auth/openid/login
- OAuth grant types
Authorization code grant
- OpenID Connect scopes
email
openid
profile
Relevant information:
I’ve spent around 24 hours of pure time for this issue and here are my findings:
- Looks like we have to added manually CA cert to java’s cacerts - at least after that I’vegot rid off TLS error in my logs
- custom keystore is here with all certs.
- Somehow I was able to login after second try - login with cognito, got redirected to login page and agter second click on “Login with SSO” I was logged in. I managed to catch the difference and looks like that referrer was different, logs are here. first is 200 and the second one is 401:
{"type":"response","@timestamp":"2023-04-04T07:07:37Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/api/v1/configuration/account","method":"get","headers":{"x-forwarded-for":"31.138.12.53","x-forwarded-proto":"h
ttps","x-forwarded-port":"443","host":"BASE_URL","x-amzn-trace-id":"Root=1-642bccb9-537b1b42cc3b58","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0","accept":"*/*",
"accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://BASE_URL/app/home","content-type":"application/json","osd-version":"2.6.0","dnt":"1","sec-fetch-dest":"empty","sec-fetch-mode":
"cors","sec-fetch-site":"same-origin"},"remoteAddress":"192.168.3.167","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0","referer":"https://BASE_URL/app/home"},"res":{"statusCo
de":200,"responseTime":10,"contentLength":9},"message":"GET /api/v1/configuration/account 200 10ms - 9.0B"}
{"type":"response","@timestamp":"2023-04-04T07:10:16Z","tags":[],"pid":1,"method":"get","statusCode":401,"req":{"url":"/api/v1/configuration/account","method":"get","headers":{"x-forwarded-for":"31.138.12.53","x-forwarded-proto":
"https","x-forwarded-port":"443","host":"BASE_URL","x-amzn-trace-id":"Root=1-642bcd58-0b4b4b4439c56ed","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0","accept":"*
/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://BASE_URL/app/login?","content-type":"application/json","osd-version":"2.6.0","dnt":"1","sec-fetch-dest":"empty","sec-fetc
h-mode":"cors","sec-fetch-site":"same-origin"},"remoteAddress":"192.168.1.125","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0","referer":"https://BASE_URL/app/login?"},"res
":{"statusCode":401,"responseTime":1,"contentLength":9},"message":"GET /api/v1/configuration/account 401 1ms - 9.0B"}
FWIW my adding to docker’s entrypoint:
function setupKeystore {
keytool -import -storetype PKCS12 -keystore os.pkc -file /usr/share/opensearch/config/admin.pem -alias admin.pem -storepass changeit -noprompt
keytool -import -storetype PKCS12 -keystore os.pkc -file /usr/share/opensearch/config/root-ca.pem -alias root-ca -storepass changeit -noprompt
keytool -import -storetype PKCS12 -keystore os.pkc -file /usr/share/opensearch/config/esnode.pem -alias esnode -storepass changeit -noprompt
keytool -import -storetype PKCS12 -keystore os.pkc -file /usr/share/opensearch/config/cognito-general.pem -alias cognito-general -storepass changeit -noprompt
keytool -import -storetype PKCS12 -keystore os.pkc -file /usr/share/opensearch/config/cognito-pool.pem -alias cognito-pool -storepass changeit -noprompt
keytool -import -cacerts -file /usr/share/opensearch/config/root-ca.pem -alias our_root -noprompt -storepass changeit
keytool -import -cacerts -file /usr/share/opensearch/config/cognito-pool.pem -alias cog-pool -noprompt -storepass changeit
keytool -import -cacerts -file /usr/share/opensearch/config/cognito-general.pem -alias co-gen -noprompt -storepass changeit
cp os.pkc /usr/share/opensearch/config/
rm os.pkc
}
Once container is started, I’ve tried to use securityadmin to apply changes (thohugh I guess it should be done on startup automcatically since exec into a pod is not a thing we should do):
~/plugins/opensearch-security/tools/securityadmin.sh -cert config/admin.pem -key config/admin.key -cacert config/root-ca.pem --disable-host-name-verification -cd ./config
And output shown that config.yml has been applied.
Also I’ve tried to dele opendistro-security index and recreate it with securityadmin.sh
Any thoughts?
OpenID documentation and information it’s a complete disaster, will be very happy to sort it once manage it.