Hi, facing issues to access and authenticate to kibana. opendistro_security plugin.
Accessing kibana UI endpoint I’m getting following error:
{"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}
Log from kibana container:
{"type":"response","@timestamp":"2021-10-22T08:10:22Z","tags":[],"pid":13,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"kibana-spc2385a.prod.spock.bmwgroup.net","sec-ch-ua":"\"\\\\Not;A\\\"Brand\";v=\"99\", \"Google Chrome\";v=\"85\", \"Chromium\";v=\"85\"","sec-ch-ua-mobile":"?0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-proto":"https, https","x-forwarded-for":"XX.XX.XX.XX, YY.YY.YY.YY","x-forwarded-port":"443","connection":"close"},"remoteAddress":"ZZ.ZZ.ZZ.ZZ","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"GET / 302 1ms - 9.0B"}
{"type":"response","@timestamp":"2021-10-22T08:10:23Z","tags":["api"],"pid":13,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"accept":"application/json, text/plain, */*","user-agent":"axios/0.21.1","host":"127.0.0.1:5601","connection":"close"},"remoteAddress":"127.0.0.1","userAgent":"axios/0.21.1"},"res":{"statusCode":200,"responseTime":1,"contentLength":9},"message":"GET /api/status 200 1ms - 9.0B"}
{"type":"response","@timestamp":"2021-10-22T08:10:23Z","tags":[],"pid":13,"method":"get","statusCode":200,"req":{"url":"/_prometheus/metrics","method":"get","headers":{"host":"9.0.13.7:5601","user-agent":"Go-http-client/1.1","accept":"application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited;q=0.7,text/plain;version=0.0.4;q=0.3","accept-encoding":"gzip","connection":"close"},"remoteAddress":"9.0.13.1","userAgent":"Go-http-client/1.1"},"res":{"statusCode":200,"responseTime":4,"contentLength":9},"message":"GET /_prometheus/metrics 200 4ms - 9.0B"}
{"type":"response","@timestamp":"2021-10-22T08:10:23Z","tags":[],"pid":13,"method":"get","statusCode":401,"req":{"url":"/auth/openid/login?nextUrl=%2F","method":"get","headers":{"host":"kibana-spc2385a.some.domain","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","sec-ch-ua":"\"\\\\Not;A\\\"Brand\";v=\"99\", \"Google Chrome\";v=\"85\", \"Chromium\";v=\"85\"","sec-ch-ua-mobile":"?0","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-proto":"https, https","x-forwarded-for":"10.84.80.136, 10.8.142.166","x-forwarded-port":"443","connection":"close"},"remoteAddress":"44.128.0.21","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"},"res":{"statusCode":401,"responseTime":0,"contentLength":9},"message":"GET /auth/openid/login?nextUrl=%2F 401 0ms - 9.0B"}
{"type":"response","@timestamp":"2021-10-22T08:10:24Z","tags":[],"pid":13,"method":"get","statusCode":401,"req":{"url":"/favicon.ico","method":"get","headers":{"host":"kibana-spc2385a.some.domain","sec-ch-ua":"\"\\\\Not;A\\\"Brand\";v=\"99\", \"Google Chrome\";v=\"85\", \"Chromium\";v=\"85\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","accept":"image/avif,image/webp,image/apng,image/*,*/*;q=0.8","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"image","referer":"https://kibana-spc2385a.some.domain/auth/openid/login?nextUrl=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-proto":"https, https","x-forwarded-for":"10.84.80.136, 10.8.142.166","x-forwarded-port":"443","connection":"close"},"remoteAddress":"44.128.0.21","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","referer":"https://kibana-spc2385a.some.domain/auth/openid/login?nextUrl=%2F"},"res":{"statusCode":401,"responseTime":1,"contentLength":9},"message":"GET /favicon.ico 401 1ms - 9.0B"}
{"type":"response","@timestamp":"2021-10-22T08:10:34Z","tags":["api"],"pid":13,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"127.0.0.1:5601","user-agent":"curl/7.66.0","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.66.0"},"res":{"statusCode":200,"responseTime":2,"contentLength":9},"message":"GET /api/status 200 2ms - 9.0B"}
{"type":"response","@timestamp":"2021-10-22T08:11:04Z","tags":["api"],"pid":13,"method":"get","statusCode":200,"req":{"url":"/api/status","method":"get","headers":{"host":"127.0.0.1:5601","user-agent":"curl/7.66.0","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"curl/7.66.0"},"res":{"statusCode":200,"responseTime":2,"contentLength":9},"message":"GET /api/status 200 2ms - 9.0B"}
The elastic stack contain following nodes :
“config-0”,
“coordinator-0”,
“coordinator-1”,
“data-0”,
“data-1”,
“data-2”,
“exporter-0”,
“ingest-0”,
“master-0”,
“master-1”,
“master-2”
Following is log from coordinator node:
Installing plugin opendistro-job-scheduler first as it is a dependency for other plugins
-> Installing file:///var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-job-scheduler-1.13.0.0.zip
-> Downloading file:///var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-job-scheduler-1.13.0.0.zip
-> Installed opendistro-job-scheduler
Installing plugin: /var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-alerting-1.13.1.0.zip
-> Installing file:///var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-alerting-1.13.1.0.zip
-> Downloading file:///var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-alerting-1.13.1.0.zip
-> Installed opendistro-alerting
Installing plugin: /var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-anomaly-detection-1.13.0.0.zip
-> Installing file:///var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-anomaly-detection-1.13.0.0.zip
-> Downloading file:///var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-anomaly-detection-1.13.0.0.zip
-> Installed opendistro-anomaly-detection
Installing plugin: /var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-index-management-1.13.2.0.zip
-> Installing file:///var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-index-management-1.13.2.0.zip
-> Downloading file:///var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-index-management-1.13.2.0.zip
-> Installed opendistro-index-management
Installing plugin: /var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-security-1.13.1.0.zip
-> Installing file:///var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-security-1.13.1.0.zip
-> Downloading file:///var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-security-1.13.1.0.zip
-> Installed opendistro_security
Installing plugin: /var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-sql-1.13.2.0.zip
-> Installing file:///var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-sql-1.13.2.0.zip
-> Downloading file:///var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/opendistro-sql-1.13.2.0.zip
-> Installed opendistro-sql
[2021-10-22T08:06:30,265][INFO ][o.e.n.Node ] [coordinator-0-node] version[7.10.2], pid[6], build[oss/tar/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.19.145-flatcar/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/13.0.2/13.0.2+8]
[2021-10-22T08:06:30,267][INFO ][o.e.n.Node ] [coordinator-0-node] JVM home [/var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/jdk-13.0.2], using bundled JDK [false]
[2021-10-22T08:06:30,267][INFO ][o.e.n.Node ] [coordinator-0-node] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms16384m, -Xmx16384m, -XX:+UseG1GC, -XX:InitiatingHeapOccupancyPercent=75, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-15288787256744784217, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:MaxDirectMemorySize=8589934592, -Des.path.home=/var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/elasticsearch-7.10.2, -Des.path.conf=/var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/elasticsearch-7.10.2/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2021-10-22T08:06:31,338][INFO ][c.a.o.s.s.t.OpenDistroSSLConfig] [coordinator-0-node] SSL dual mode is disabled
[2021-10-22T08:06:31,339][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [coordinator-0-node] ES Config path is /var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/elasticsearch-7.10.2/config
[2021-10-22T08:06:31,548][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [coordinator-0-node] JVM supports TLSv1.3
[2021-10-22T08:06:31,549][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [coordinator-0-node] Config directory is /var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/elasticsearch-7.10.2/config/, from there the key- and truststore files are resolved relatively
[2021-10-22T08:06:31,553][INFO ][c.a.o.s.s.u.SSLCertificateHelper] [coordinator-0-node] No alias given, use the first one: default
[2021-10-22T08:06:31,561][WARN ][c.a.o.s.s.u.SSLCertificateHelper] [coordinator-0-node] Certificate chain for alias default contains a root certificate
[2021-10-22T08:06:31,876][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [coordinator-0-node] HTTPS client auth mode OPTIONAL
[2021-10-22T08:06:31,877][INFO ][c.a.o.s.s.u.SSLCertificateHelper] [coordinator-0-node] No alias given, use the first one: default
[2021-10-22T08:06:31,877][WARN ][c.a.o.s.s.u.SSLCertificateHelper] [coordinator-0-node] Certificate chain for alias default contains a root certificate
[2021-10-22T08:06:31,968][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [coordinator-0-node] TLS Transport Client Provider : JDK
[2021-10-22T08:06:31,968][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [coordinator-0-node] TLS Transport Server Provider : JDK
[2021-10-22T08:06:31,968][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [coordinator-0-node] TLS HTTP Provider : JDK
[2021-10-22T08:06:31,968][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [coordinator-0-node] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2, TLSv1.1]
[2021-10-22T08:06:31,969][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [coordinator-0-node] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2, TLSv1.1]
[2021-10-22T08:06:32,145][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [coordinator-0-node] Clustername: infra__logging__elastic
[2021-10-22T08:06:32,149][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [coordinator-0-node] Directory /var/lib/mesos/slave/slaves/91fd3984-0390-4800-9e53-840967dd4fa6-S11/frameworks/28730a4a-181c-42dd-8bff-e4decd646071-0000/executors/infra.logging.elastic__coordinator__0abd39af-b21a-4b8e-bb82-234a9ce343cc/runs/bda75ea3-7c30-4cdf-8438-28f47404636f/containers/9124f4be-92a2-4084-9e67-fad5a0aa7fb3/elasticsearch-7.10.2/config has insecure file permissions (should be 0700)
[2021-10-22T08:06:32,429][INFO ][c.a.o.j.JobSchedulerPlugin] [coordinator-0-node] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2021-10-22T08:06:32,433][INFO ][c.a.o.j.JobSchedulerPlugin] [coordinator-0-node] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2021-10-22T08:06:32,435][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [aggs-matrix-stats]
[2021-10-22T08:06:32,435][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [analysis-common]
[2021-10-22T08:06:32,436][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [geo]
[2021-10-22T08:06:32,436][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [ingest-common]
[2021-10-22T08:06:32,436][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [ingest-geoip]
[2021-10-22T08:06:32,436][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [ingest-user-agent]
[2021-10-22T08:06:32,436][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [kibana]
[2021-10-22T08:06:32,436][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [lang-expression]
[2021-10-22T08:06:32,436][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [lang-mustache]
[2021-10-22T08:06:32,437][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [lang-painless]
[2021-10-22T08:06:32,437][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [mapper-extras]
[2021-10-22T08:06:32,437][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [parent-join]
[2021-10-22T08:06:32,437][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [percolator]
[2021-10-22T08:06:32,437][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [rank-eval]
[2021-10-22T08:06:32,437][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [reindex]
[2021-10-22T08:06:32,438][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [repository-url]
[2021-10-22T08:06:32,438][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded module [transport-netty4]
[2021-10-22T08:06:32,438][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded plugin [opendistro-alerting]
[2021-10-22T08:06:32,438][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded plugin [opendistro-anomaly-detection]
[2021-10-22T08:06:32,438][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded plugin [opendistro-index-management]
[2021-10-22T08:06:32,439][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded plugin [opendistro-job-scheduler]
[2021-10-22T08:06:32,439][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded plugin [opendistro-sql]
[2021-10-22T08:06:32,439][INFO ][o.e.p.PluginsService ] [coordinator-0-node] loaded plugin [opendistro_security]
[2021-10-22T08:06:32,466][INFO ][o.e.e.NodeEnvironment ] [coordinator-0-node] using [1] data paths, mounts [[/ (/dev/sda9)]], net usable_space [389.3gb], net total_space [430.8gb], types [ext4]
[2021-10-22T08:06:32,467][INFO ][o.e.e.NodeEnvironment ] [coordinator-0-node] heap size [16gb], compressed ordinary object pointers [true]
[2021-10-22T08:06:32,530][INFO ][o.e.n.Node ] [coordinator-0-node] node name [coordinator-0-node], node ID [v8bOKwzsR4OfwsTztoofJA], cluster name [infra__logging__elastic], roles [remote_cluster_client]
[2021-10-22T08:06:34,439][WARN ][c.a.o.s.c.Salt ] [coordinator-0-node] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2021-10-22T08:06:34,457][ERROR][c.a.o.s.a.s.SinkProvider ] [coordinator-0-node] Default endpoint could not be created, auditlog will not work properly.
[2021-10-22T08:06:34,457][WARN ][c.a.o.s.a.r.AuditMessageRouter] [coordinator-0-node] No default storage available, audit log may not work properly. Please check configuration.
[2021-10-22T08:06:34,458][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Message routing enabled: false
[2021-10-22T08:06:34,484][INFO ][c.a.o.s.f.OpenDistroSecurityFilter] [coordinator-0-node] <NONE> indices are made immutable.
[2021-10-22T08:06:34,760][INFO ][c.a.o.a.b.ADCircuitBreakerService] [coordinator-0-node] Registered memory breaker.
[2021-10-22T08:06:34,952][INFO ][o.e.t.NettyAllocator ] [coordinator-0-node] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=8mb}]
[2021-10-22T08:06:35,012][INFO ][o.e.d.DiscoveryModule ] [coordinator-0-node] using discovery type [zen] and seed hosts providers [settings]
[2021-10-22T08:06:35,257][WARN ][o.e.g.DanglingIndicesState] [coordinator-0-node] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-10-22T08:06:35,484][INFO ][o.e.n.Node ] [coordinator-0-node] initialized
[2021-10-22T08:06:35,484][INFO ][o.e.n.Node ] [coordinator-0-node] starting ...
[2021-10-22T08:06:35,575][INFO ][o.e.t.TransportService ] [coordinator-0-node] publish_address {coordinator-0-node.infraloggingelastic.autoip.dcos.thisdcos.directory/9.0.12.9:1026}, bound_addresses {[::1]:1026}, {127.0.0.1:1026}, {9.0.12.9:1026}
[2021-10-22T08:06:35,656][INFO ][o.e.b.BootstrapChecks ] [coordinator-0-node] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-10-22T08:06:36,125][INFO ][o.e.c.s.ClusterApplierService] [coordinator-0-node] master node changed {previous [], current [{master-2-node}{Ol1A4joDR_W48PwgLR63QQ}{9h300XnCTJy0Vda5Zod2Gw}{master-2-node.infraloggingelastic.autoip.dcos.thisdcos.directory}{9.0.6.9:9200}{mr}{zone=onprem-1a}]}, added {{data-1-node}{i_XcTo2vS7KykTClakWR6g}{5Dj5norySOuQvHQs9cpJKA}{data-1-node.infraloggingelastic.autoip.dcos.thisdcos.directory}{9.0.14.5:1026}{dr}{zone=onprem-1a},{master-1-node}{YIrZvablRhSQQIw3e4aplg}{3sF1GKgoRFOSexcIFqZ4Vw}{master-1-node.infraloggingelastic.autoip.dcos.thisdcos.directory}{9.0.12.6:9200}{mr}{zone=onprem-1a},{data-2-node}{mDmCnnikRMmAQeA1RxZ95g}{VG3Ds9xmQPO0_nuzs3tx4g}{data-2-node.infraloggingelastic.autoip.dcos.thisdcos.directory}{9.0.12.4:1026}{dr}{zone=onprem-1a},{master-0-node}{icFL7pM9Q0qFZ4gwy03hYQ}{mx9ZM95qTlipR9lEyO9E-g}{master-0-node.infraloggingelastic.autoip.dcos.thisdcos.directory}{9.0.18.5:9200}{mr}{zone=onprem-1a},{coordinator-1-node}{f2zB01-tTpyjgpGN-3ko0Q}{KI1tjRj0RwWQUicANRgVeA}{coordinator-1-node.infraloggingelastic.autoip.dcos.thisdcos.directory}{9.0.7.8:1026}{r}{zone=onprem-1a},{data-0-node}{sTAdh9I3R6-hhoDcnOPKvg}{gfOFspN0Tm6S2if0wh4ssw}{data-0-node.infraloggingelastic.autoip.dcos.thisdcos.directory}{9.0.3.4:1026}{dr}{zone=onprem-1a},{ingest-0-node}{nRcWds-QTrOYXC-zjuvCBw}{I50c1E8jR7mQ0XSJjOltoA}{ingest-0-node.infraloggingelastic.autoip.dcos.thisdcos.directory}{9.0.11.7:1026}{ir}{zone=onprem-1a},{master-2-node}{Ol1A4joDR_W48PwgLR63QQ}{9h300XnCTJy0Vda5Zod2Gw}{master-2-node.infraloggingelastic.autoip.dcos.thisdcos.directory}{9.0.6.9:9200}{mr}{zone=onprem-1a}}, term: 7, version: 470930, reason: ApplyCommitRequest{term=7, version=470930, sourceNode={master-2-node}{Ol1A4joDR_W48PwgLR63QQ}{9h300XnCTJy0Vda5Zod2Gw}{master-2-node.infraloggingelastic.autoip.dcos.thisdcos.directory}{9.0.6.9:9200}{mr}{zone=onprem-1a}}
[2021-10-22T08:06:36,589][INFO ][o.e.h.AbstractHttpServerTransport] [coordinator-0-node] publish_address {coordinator-0-node.infraloggingelastic.autoip.dcos.thisdcos.directory/9.0.12.9:1025}, bound_addresses {[::1]:1025}, {127.0.0.1:1025}, {9.0.12.9:1025}
[2021-10-22T08:06:36,589][INFO ][o.e.n.Node ] [coordinator-0-node] started
[2021-10-22T08:06:36,590][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [coordinator-0-node] Node started
[2021-10-22T08:06:36,590][INFO ][c.a.o.s.c.ConfigurationRepository] [coordinator-0-node] Will not attempt to create index .opendistro_security and default configs if they are absent. Use securityadmin to initialize cluster
[2021-10-22T08:06:36,591][INFO ][c.a.o.s.c.ConfigurationRepository] [coordinator-0-node] Background init thread started. Install default config?: false
[2021-10-22T08:06:36,591][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [coordinator-0-node] 0 Open Distro Security modules loaded so far: []
[2021-10-22T08:06:36,764][INFO ][stdout ] [coordinator-0-node] [FINE] No subscribers registered for event class com.amazon.opendistroforelasticsearch.security.securityconf.DynamicConfigFactory$NodesDnModelImpl
[2021-10-22T08:06:36,765][INFO ][stdout ] [coordinator-0-node] [FINE] No subscribers registered for event class org.greenrobot.eventbus.NoSubscriberEvent
[2021-10-22T08:06:36,766][INFO ][c.a.o.s.c.ConfigurationRepository] [coordinator-0-node] Hot-reloading of audit configuration is disabled. Using configuration with defaults from elasticsearch settings. Populate the configuration in index using audit.yml or securityadmin to enable it.
[2021-10-22T08:06:36,766][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Auditing on REST API is enabled.
[2021-10-22T08:06:36,766][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing.
[2021-10-22T08:06:36,766][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Auditing on Transport API is enabled.
[2021-10-22T08:06:36,766][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing.
[2021-10-22T08:06:36,767][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Auditing of request body is enabled.
[2021-10-22T08:06:36,767][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Bulk requests resolution is disabled during request auditing.
[2021-10-22T08:06:36,767][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Index resolution is enabled during request auditing.
[2021-10-22T08:06:36,767][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Sensitive headers auditing is enabled.
[2021-10-22T08:06:36,767][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Auditing requests from kibanaserver users is disabled.
[2021-10-22T08:06:36,767][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Auditing of external configuration is disabled.
[2021-10-22T08:06:36,767][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Auditing of internal configuration is disabled.
[2021-10-22T08:06:36,767][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Auditing only metadata information for read request is disabled.
[2021-10-22T08:06:36,768][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Auditing will watch {} for read requests.
[2021-10-22T08:06:36,768][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Auditing read operation requests from kibanaserver users is disabled.
[2021-10-22T08:06:36,768][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Auditing only metadata information for write request is disabled.
[2021-10-22T08:06:36,768][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Auditing diffs for write requests is disabled.
[2021-10-22T08:06:36,768][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Auditing write operation requests from kibanaserver users is disabled.
[2021-10-22T08:06:36,768][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Auditing will watch <NONE> for write requests.
[2021-10-22T08:06:36,768][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] .opendistro_security is used as internal security index.
[2021-10-22T08:06:36,768][INFO ][c.a.o.s.a.i.AuditLogImpl ] [coordinator-0-node] Internal index used for posting audit logs is null
[2021-10-22T08:06:36,768][INFO ][c.a.o.s.c.ConfigurationRepository] [coordinator-0-node] Node 'coordinator-0-node' initialized
[2021-10-22T08:06:39,234][ERROR][c.a.o.s.s.h.n.OpenDistroSecuritySSLNettyHttpServerTransport] [coordinator-0-node] Exception during establishing a SSL connection: java.net.SocketException: Connection reset
java.net.SocketException: Connection reset
at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:345) ~[?:?]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:376) ~[?:?]
at org.elasticsearch.transport.CopyBytesSocketChannel.readFromSocketChannel(CopyBytesSocketChannel.java:141) ~[transport-netty4-client-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.CopyBytesSocketChannel.doReadBytes(CopyBytesSocketChannel.java:126) ~[transport-netty4-client-7.10.2.jar:7.10.2]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:830) [?:?]```
Any ideas / hints where to search for a root cause or how t get this working ?
[details="Summary"]
This text will be hidden
[/details]
Thank you for your support in advance.