# Kibana on k8s cannot connect to elasticsearch

**URL:** <https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130>\
**Category:** Open Source Elasticsearch and Kibana\
**Created:** [February 26, 2021, 7:36am UTC](https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130 "2021-02-26T07:36:59Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![johnmops](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/johnmops/32/1275_2.png) [@johnmops](https://forum.opensearch.org/u/johnmops)\
**Post date:** [February 26, 2021, 7:36am UTC](https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130/1 "2021-02-26T07:36:59Z")

</div>

Hello guys,

I am having an issue with integrating Kiabana with Okta OpenID.

1. Kibana config:

```auto
opendistro_security.auth.type: "openid"
opendistro_security.openid.connect_url: "https://my_domain/.well-known/openid-configuration"
opendistro_security.openid.client_id: "0oa9dl1e4VRStlugu5d6"
opendistro_security.openid.client_secret: "my_key"
server.name: kibana
server.host: "0"
elasticsearch.hosts: https://localhost:9200
elasticsearch.ssl.verificationMode: none
elasticsearch.username: kibanaserver
elasticsearch.password: kibanaserver
elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization"]

opendistro_security.multitenancy.enabled: true
opendistro_security.multitenancy.tenants.preferred: ["Private", "Global"]
opendistro_security.readonly_mode.roles: ["kibana_read_only"]

# Use this setting if you are running kibana without https
opendistro_security.cookie.secure: false

newsfeed.enabled: false
telemetry.optIn: false
telemetry.enabled: false
security.showInsecureClusterWarning: false

```

1. Elasticsearch config.yaml:

```auto
_meta:
type: "config"
config_version: 2

config:
dynamic:
  # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
  # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
  # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
  #filtered_alias_mode: warn
  #do_not_fail_on_forbidden: false
  #kibana:
  # Kibana multitenancy
  #multitenancy_enabled: true
  #server_username: kibanaserver
  #index: '.kibana'
  http:
    anonymous_auth_enabled: false
    xff:
      enabled: false
      internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
      #internalProxies: '.*' # trust all internal proxies, regex pattern
      #remoteIpHeader: 'x-forwarded-for'
      ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
      ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
      ###### and here https://tools.ietf.org/html/rfc7239
      ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
  authc:
    basic_internal_auth_domain:
      description: "Authenticate via HTTP Basic against internal users database"
      http_enabled: true
      transport_enabled: true
      order: 0
      http_authenticator:
        type: basic
        challenge: false
      authentication_backend:
        type: internal
    openid_auth_domain:
      http_enabled: true
      transport_enabled: true
      order: 1
      http_authenticator:
        type: openid
        challenge: false
        config:
          subject_key: preffered_username
          roles_key: roles
          openid_connect_url: https://my_domain/.well-known/openid-configuration
      authentication_backend:
        type: noop

```

1. Internal\_useras.yml: basic demo users

\_meta:

```auto
type: "internalusers"
config_version: 2

new-user:
hash: "$2y$12$88IFVl6IfIwCFh5aQYfOmuXVL9j2hz/GusQb35o.4sdTDAEMTOD.K"
reserved: false
hidden: false
opendistro_security_roles:
- "some-security-role"
backend_roles:
- "some-backend-role"
attributes:
  attribute1: "value1"
static: false

admin:
hash: "$2a$12$VcCDgh2NDk07JGN0rjGbM.Ad41qVR/YFJcgHp0UGns5JDymv..TOG"
reserved: true
backend_roles:
- "admin"
description: "Demo admin user"

kibanaserver:
hash: "$2a$12$4AcgAt3xwOWadA5s5blL6ev39OXDNhmOesEoo33eZtrq2N0YrU3H."
reserved: true
description: "Demo kibanaserver user"

```

- Issue:

As soon as i am installing the helm chart after adding the openid settings i get this error in Kibana none stop:

{“type”:“log”,“@timestamp”:“2021-02-25T16:25:30Z”,“tags”:[“error”,“elasticsearch”,“data”],“pid”:1,“message”:“[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200”}

The above prevents the server from going up.

- If i am reverting back to regular authentication without the openid, everything works fine.
- Tried to switch the hosts to “http”
- Tried to switch the openid auth in config.yaml to “0”
- Tried to inject an elasticearch url as per the kubernetes FQDN
- Tried different users besides “kibanaserver”
- Played around with multiple different configuration in kibana.yml

I tried vertuallye everything for the past 7 hours and i do hope i am missing someting here.

Thank you for yout time.

---

<div class="post-metadata">

**Author:** ![johnmops](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/johnmops/32/1275_2.png) [@johnmops](https://forum.opensearch.org/u/johnmops)\
**Post date:** [February 27, 2021, 9:00am UTC](https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130/2 "2021-02-27T09:00:17Z")

</div>

Something else that is extremely strange.

If i am installing the default helm chart, kibana is working and connects to elastcsearch.  
If i take the exact same config from the default kibana.yml and paste it to the values.yml under kibana.config, the same error is happening.

---

<div class="post-metadata">

**Author:** ![mmamaenko](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@mmamaenko](https://forum.opensearch.org/u/mmamaenko)\
**Post date:** [February 27, 2021, 5:11pm UTC](https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130/3 "2021-02-27T17:11:15Z")

</div>

why are you using **elasticsearch.hosts: [https://localhost:9200](https://localhost:9200)** in kibana config? Are you running kibana and elasticsearch on the same pod? Sorry, did not use helm to install.

---

<div class="post-metadata">

**Author:** ![johnmops](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/johnmops/32/1275_2.png) [@johnmops](https://forum.opensearch.org/u/johnmops)\
**Post date:** [February 27, 2021, 5:22pm UTC](https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130/4 "2021-02-27T17:22:50Z")

</div>

there is an iptable that redirects it to the host name of the releval elastisearch client service.  
The issue now is different:

Redirects to Okta succesffuly, but getting an error in the kibana pod

{“type”:“log”,“@timestamp”:“2021-02-27T16:00:38Z”,“tags”:[“error”,“plugins”,“opendistroSecurityKibana”],“pid”:1,“message”:“OpenId authentication failed: Error: Authentication Exception”}

---

<div class="post-metadata">

**Author:** ![mmamaenko](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@mmamaenko](https://forum.opensearch.org/u/mmamaenko)\
**Post date:** [February 27, 2021, 6:23pm UTC](https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130/5 "2021-02-27T18:23:50Z")

</div>

how jwt from okta look like?

---

<div class="post-metadata">

**Author:** ![johnmops](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/johnmops/32/1275_2.png) [@johnmops](https://forum.opensearch.org/u/johnmops)\
**Post date:** [February 27, 2021, 6:35pm UTC](https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130/6 "2021-02-27T18:35:42Z")

</div>

i am not using tls or the token. Just the preffered\_username.

---

<div class="post-metadata">

**Author:** ![mmamaenko](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@mmamaenko](https://forum.opensearch.org/u/mmamaenko)\
**Post date:** [February 27, 2021, 6:41pm UTC](https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130/7 "2021-02-27T18:41:51Z")

</div>

after kibana redirects user to OpenIDP, user provides username and password, OpenID provider checked credentials and generate token with result and some additional info and sends it back to kibana. This is how Auth0 or Keycloak are communicating with kibana, I am sure Okta is working the same way

---

<div class="post-metadata">

**Author:** ![johnmops](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/johnmops/32/1275_2.png) [@johnmops](https://forum.opensearch.org/u/johnmops)\
**Post date:** [February 27, 2021, 7:03pm UTC](https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130/8 "2021-02-27T19:03:59Z")

</div>

Hmm any idea where can i check the token that was generated?

---

<div class="post-metadata">

**Author:** ![mmamaenko](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@mmamaenko](https://forum.opensearch.org/u/mmamaenko)\
**Post date:** [February 27, 2021, 8:15pm UTC](https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130/9 "2021-02-27T20:15:07Z")

</div>

[https://oidcdebugger.com/](https://oidcdebugger.com/) or you can find browser addons for jwt debugging. Also do you have any logs from Okta side about authentication? I remember that for instance Auth0 shows log for authentication

---

<div class="post-metadata">

**Author:** ![johnmops](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/johnmops/32/1275_2.png) [@johnmops](https://forum.opensearch.org/u/johnmops)\
**Post date:** [February 28, 2021, 6:01am UTC](https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130/10 "2021-02-28T06:01:33Z")

</div>

![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/2X/9/9503d787623ac29165afb08ed4313dd37ca382fd.png)

Okta side is perfectly fine.

In kibana pod the only error i see is the one i already sent

---

<div class="post-metadata">

**Author:** ![mmamaenko](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@mmamaenko](https://forum.opensearch.org/u/mmamaenko)\
**Post date:** [February 28, 2021, 6:39am UTC](https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130/11 "2021-02-28T06:39:02Z")

</div>

Can you see token generated in Okta? Also you can increase log level to DEBUG in elasticsearch/kibana to have more information.

---

<div class="post-metadata">

**Author:** ![johnmops](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/johnmops/32/1275_2.png) [@johnmops](https://forum.opensearch.org/u/johnmops)\
**Post date:** [February 28, 2021, 6:51am UTC](https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130/12 "2021-02-28T06:51:13Z")

</div>

Something i had not noticed before. From elasticsearch:

[2021-02-28T06:50:38,136][WARN][c.a.o.s.h.HTTPBasicAuthenticator] [od-es-opendistro-es-master-0] No ‘Basic Authorization’ header, send 401 and ‘WWW-Authenticate Basic’

[2021-02-28T06:50:38,136][INFO][c.a.d.a.h.j.k.JwtVerifier] [od-es-opendistro-es-master-0] Escaped Key ID from JWT Token

[2021-02-28T06:50:38,137][WARN][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [od-es-opendistro-es-master-0] Failed to get subject from JWT claims, check if subject\_key ‘preffered\_username’ is correct.

[2021-02-28T06:50:38,137][ERROR][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [od-es-opendistro-es-master-0] No subject found in JWT token

[2021-02-28T06:50:38,137][WARN][c.a.o.s.a.BackendRegistry] [od-es-opendistro-es-master-0] Authentication finally failed for null from 10.64.138.3:38886

---

<div class="post-metadata">

**Author:** ![johnmops](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/johnmops/32/1275_2.png) [@johnmops](https://forum.opensearch.org/u/johnmops)\
**Post date:** [February 28, 2021, 7:02am UTC](https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130/13 "2021-02-28T07:02:38Z")

</div>

I am an idiot. I mispelled preferred\_username…  
Thank you very much for your replies

---

<div class="post-metadata">

**Author:** ![boris.uzhinskiy](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/boris.uzhinskiy/32/1093_2.png) [@boris.uzhinskiy](https://forum.opensearch.org/u/boris.uzhinskiy)\
**Post date:** [July 21, 2021, 4:19pm UTC](https://forum.opensearch.org/t/kibana-on-k8s-cannot-connect-to-elasticsearch/5130/14 "2021-07-21T16:19:53Z")

</div>

Hi @johnmops  
Which “preferred\_username” you means?