Kibana security plugin /api/status is green eventhough IdP endpoint is not reachable

[chaitra]

Hi,
I am using OD security 1.13.x and ES 7.10.2 rpm.

Error: Failed when trying to obtain the endpoints from your IdP

I see that inspite of above error, Kibana security endpoint is green.

			"id": "plugin:opendistroSecurityKibana@7.10.2",
			"message": "All dependencies are available",
			"since": "2021-07-29T12:51:53.485Z",
			"state": "green",
			"icon": "success",
			"uiColor": "secondary"

Previously I was using OD security 1.9.x but in that the plugin status would turn red if IdP endpoint is not reachable.

Let me know how to get proper status output for security plugin.

[pablo]

@chaitra

Kibana security endpoint would turn red if it was misconfigured. (i.e. missing options or incorrect value format).

Could you share config.yml, kibana.yml and elastcisearch.yml files?

[chaitra]

Hi,
Inspite of Error: Failed when trying to obtain the endpoints from your IdP in kibana, security plugin status is green.

Error log:

Error: connect ECONNREFUSED x.x.x.x:32788\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)\n  errno: 'ECONNREFUSED',\n  code: 'ECONNREFUSED',\n  syscall: 'connect',\n  address: 'x.x.x.x',\n  port: 32788,\n  trace:\n   [ { method: 'GET',\n       url:\n        'https://<keycloak-url>:32788/access/realms/chaitra/.well-known/openid-configuration' } ],\n  isBoom: true,\n  isServer: true,\n  data: null,\n  output:\n   { statusCode: 502,\n     payload:\n      { message: 'Client request error: connect ECONNREFUSED x.x.x.x:32788',\n        statusCode: 502,\n        error: 'Bad Gateway' },\n     headers: {} }
Detected an unhandled Promise rejection.\nError: Failed when trying to obtain the endpoints from your IdP"

Configurations are as below:

_meta:
  type: "config"
  config_version: 2
config:
  dynamic:
    kibana:
       multitenancy_enabled: false
       server_username: kibanaserver
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: ".+"
    authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: "basic"
          challenge: false   
          config: {}
        authentication_backend:
          type: "intern"
          config: {}
      openid_auth_domain:
        http_enabled: true              
        transport_enabled: false
        order: 1
        http_authenticator:
          type: openid
          challenge: false
          config:
            subject_key: preferred_username
            roles_key: roles
            openid_connect_url: https://<keycloak-url>/access/realms/master/.well-known/openid-configuration
            openid_connect_idp:
              enable_ssl: true
              verify_hostnames: false
              trust_all: false    
              pemtrustedcas_filepath: "<path-to-ca-pem-file>"
        authentication_backend:
            type: noop

opendistro_security.auth.type: "openid"
opendistro_security.openid.connect_url: "https://<keycloak-url>/access/realms/master/.well-known/openid-configuration"
opendistro_security.openid.client_id: "test"
opendistro_security.openid.client_secret: "xxxxxxxxxxxxxx"
opendistro_security.openid.header: "Authorization"
opendistro_security.openid.base_redirect_url: "https://<kibanaurl>/<basePath>"
opendistro_security.openid.root_ca: "<path-to-ca-pem-file>"

cluster.name: my-cluster
cluster.initial_master_nodes: node-0

node.name: node-0
node.roles: "master, ingest, data"
discovery.seed_hosts: node-0

network.host: "_site_"
path.data: /data/data
path.logs: /data/log
http.compression: true

opendistro_security.ssl.transport.enable_openssl_if_available: false
opendistro_security.ssl.transport.keystore_type: JKS
opendistro_security.ssl.transport.keystore_filepath: <path-to-keystore.jks>
opendistro_security.ssl.transport.keystore_password: <password>
opendistro_security.ssl.transport.truststore_type: JKS
opendistro_security.ssl.transport.truststore_filepath: <path-to-truststore.jks>
opendistro_security.ssl.transport.truststore_password: <password>
opendistro_security.ssl.transport.enforce_hostname_verification: false

opendistro_security.ssl.http.enable_openssl_if_available: true
opendistro_security.ssl.http.clientauth_mode: OPTIONAL
opendistro_security.ssl.http.keystore_type: JKS
opendistro_security.ssl.http.keystore_filepath: <path-to-keystore.jks>
opendistro_security.ssl.http.keystore_password: <password>
opendistro_security.ssl.http.truststore_type: JKS
opendistro_security.ssl.http.truststore_filepath: <path-to-truststore.jks>
opendistro_security.ssl.http.truststore_password: <password>

opendistro_securityauthcz.admin_dn: "<auth-admin-dn>"
opendistro_securityauthcz.nodes_dn: "<nodes-dn>"

[pablo]

Hi @chaitra

According to your config and error. You’re trying to reach keycloak on port 32788.
Did you change Keycloak’s default HTTPS port 8443 to 32788?

openid_connect_url points to ‘https:///access/realms/master/.well-known/openid-configuration’
In my config, that URL is ‘https:///auth/realms/master/.well-known/openid-configuration’

To verify that, copy the URL from openid_connect_url to your browser. That should return the following output.

Do you get to Keycloak’s login screen or did you get this error before?

[chaitra]

Hi,
I m using k8s environment. I have deployed Keycloak using NodePort. So for me keycloak url is accessible via 32788.
Purposely I made keycloak service down and Kibana had above mentioned error log. Then checked OD security endpoint health using /api/status. The status says green(ideally it should have been Red).

[pablo]

@chaitra

How did you test plugin status in 1.9.0?

[pablo]

@chaitra

I can get red status on the security plug-in in ODFE 1.9 only during Kibana startup. Plug-in doesn’t turn red when keycloak is turned off after successful Kibana startup.

As per my tests, since version 1.10, Kibana will start and load security plug-in successfully even when keycloak is down. (OpenID scenario)

[chaitra]

Hi,
Yes I have tried the scenario on OD security plugin 1.9. In that during startup of Kibana, it tries to connects to keycloak endpoint and if it is not reachable plugin status turns to Red.
But the same scenario in OD security plugin 1.13.x doesn’t turn Red. Why is it so?

[pablo]

Hi @chaitra

It looks like the design change. When you look at the logs in 1.13.x you will see that connectivity test is still performed and connectivity issue reported, but the default behaviour is not to fail Kibana start-up.

As you said, the documentation doesn’t explain this behaviour. The best place to ask such a question is GitHub, where Dev can answer.

Issues · opensearch-project/security

Still, I think that the plug-in config is validated only during Kibana start-up and not actively monitored. If Keycloak will go offline then the plug-in status will remain green in both 1.9.0 and 1.13.x.

Maybe you should consider monitoring the IdP (i.e. Keycloak) with IP or API query in such a case.