**Describe the bug**
1 hour after login, Kibana will show one of these 2 errors… instead of the requested page:
* On the root domain of Kibana: `{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}`
* On all other pages: `{"statusCode":401,"error":"Unauthorized","message":"Response Error"}`
The only way to work around this issue is to delete cookies.
**To Reproduce**
Steps to reproduce the behavior:
1. Login to Kibana via a SAML Identity Provider (e.g. Okta)
2. Wait for 1 hour
3. Try to refresh/browse to a new Kibana page
4. See that Kibana shows `{"statusCode":401,"error":"Unauthorized","message":"Response Error"}` instead of showing the requested page
5. See that Kibana shows `{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}` when manually going to the root Kibana domain
Note: to create the Okta app, I followed the instructions here: [AWS - Add Single Sign-On (SSO) to Open Distro for Elasticsearch Kibana using SAML and Okta](https://aws.amazon.com/blogs/opensource/open-distro-for-elasticsearch-saml-okta/).
**Expected behavior**
The **internal JWT** created by OpenDistro (I'm not sure exactly what component creates it) **should be automatically renewed** and Kibana shouldn't throw an error (either 401 or 500) when visiting a page 1 hour or more after initial login.
**Logs**
ES Node Logs
```text
[2021-09-15T09:08:24,861][TRACE][c.a.o.s.a.BackendRegistry] [es3.logs.example.com] Rest authentication request from 10.0.3.4:32528 [original: /10.0.3.4:32528]
[2021-09-15T09:08:24,861][DEBUG][c.a.o.s.a.BackendRegistry] [es3.logs.example.com] Check authdomain for rest noop/0 or 2 in total
[2021-09-15T09:08:24,861][TRACE][c.a.o.s.a.BackendRegistry] [es3.logs.example.com] Try to extract auth creds from clientcert http authenticator
[2021-09-15T09:08:24,861][TRACE][c.a.o.s.h.HTTPClientCertAuthenticator] [es3.logs.example.com] No CLIENT CERT, send 401
[2021-09-15T09:08:24,861][TRACE][c.a.o.s.a.BackendRegistry] [es3.logs.example.com] No 'Authorization' header, send 403
[2021-09-15T09:08:24,861][DEBUG][c.a.o.s.a.BackendRegistry] [es3.logs.example.com] Check authdomain for rest noop/1 or 2 in total
[2021-09-15T09:08:24,861][TRACE][c.a.o.s.a.BackendRegistry] [es3.logs.example.com] Try to extract auth creds from saml http authenticator
[2021-09-15T09:08:24,862][INFO ][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [es3.logs.example.com] Extracting JWT token from [REDACTED JWT TOKEN HERE] failed
com.amazon.dlic.auth.http.jwt.keybyoidc.BadCredentialsException: The token has expired
at com.amazon.dlic.auth.http.jwt.keybyoidc.JwtVerifier.getVerifiedJwtToken(JwtVerifier.java:85) ~[opendistro_security-1.13.1.0.jar:1.13.1.0]
at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.extractCredentials0(AbstractHTTPJwtAuthenticator.java:108) [opendistro_security-1.13.1.0.jar:1.13.1.0]
at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.access$000(AbstractHTTPJwtAuthenticator.java:47) [opendistro_security-1.13.1.0.jar:1.13.1.0]
at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator$1.run(AbstractHTTPJwtAuthenticator.java:90) [opendistro_security-1.13.1.0.jar:1.13.1.0]
at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator$1.run(AbstractHTTPJwtAuthenticator.java:87) [opendistro_security-1.13.1.0.jar:1.13.1.0]
at java.security.AccessController.doPrivileged(AccessController.java:312) [?:?]
at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.extractCredentials(AbstractHTTPJwtAuthenticator.java:87) [opendistro_security-1.13.1.0.jar:1.13.1.0]
at com.amazon.dlic.auth.http.saml.HTTPSamlAuthenticator.extractCredentials(HTTPSamlAuthenticator.java:148) [opendistro_security-1.13.1.0.jar:1.13.1.0]
at com.amazon.opendistroforelasticsearch.security.auth.BackendRegistry.authenticate(BackendRegistry.java:421) [opendistro_security-1.13.1.0.jar:1.13.1.0]
at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter.checkAndAuthenticateRequest(OpenDistroSecurityRestFilter.java:177) [opendistro_security-1.13.1.0.jar:1.13.1.0]
at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter.access$000(OpenDistroSecurityRestFilter.java:66) [opendistro_security-1.13.1.0.jar:1.13.1.0]
at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityRestFilter$1.handleRequest(OpenDistroSecurityRestFilter.java:113) [opendistro_security-1.13.1.0.jar:1.13.1.0]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:258) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:340) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:191) [elasticsearch-7.10.2.jar:7.10.2]
at com.amazon.opendistroforelasticsearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opendistro_security-1.13.1.0.jar:1.13.1.0]
at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:319) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:384) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:309) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) [transport-netty4-client-7.10.2.jar:7.10.2]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:28) [transport-netty4-client-7.10.2.jar:7.10.2]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.10.2.jar:7.10.2]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: org.apache.cxf.rs.security.jose.jwt.JwtException: The token has expired
at org.apache.cxf.rs.security.jose.jwt.JwtUtils.validateJwtExpiry(JwtUtils.java:58) ~[cxf-rt-rs-security-jose-3.4.0.jar:3.4.0]
at com.amazon.dlic.auth.http.jwt.keybyoidc.JwtVerifier.validateClaims(JwtVerifier.java:119) ~[opendistro_security-1.13.1.0.jar:1.13.1.0]
at com.amazon.dlic.auth.http.jwt.keybyoidc.JwtVerifier.getVerifiedJwtToken(JwtVerifier.java:81) ~[opendistro_security-1.13.1.0.jar:1.13.1.0]
... 74 more
[2021-09-15T09:08:24,865][DEBUG][c.o.s.a.AuthnRequest ] [es3.logs.example.com] AuthNRequest --> <samlp:AuthnRequest xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" ID="ONELOGIN_{SOME_UUID}" Version="2.0" IssueInstant="2021-09-15T09:08:24Z" ForceAuthn="true" Destination="https://subdomain.okta.com/app/xxx/yyy/sso/saml" ProtocolBinding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" AssertionConsumerServiceURL="https://kb.logs.example.com/_opendistro/_security/saml/acs"><saml:Issuer>logs-kibana-saml</saml:Issuer><samlp:NameIDPolicy Format="urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified" AllowCreate="true" /></samlp:AuthnRequest>
[2021-09-15T09:08:24,865][TRACE][o.e.i.b.in_flight_requests] [es3.logs.example.com] [in_flight_requests] Adjusted breaker by [0] bytes, now [0]
[2021-09-15T09:08:24,866][TRACE][o.e.h.HttpTracer ] [es3.logs.example.com] [422][8027f54c-0e8c-4318-b8c8-a5fab657dd63][UNAUTHORIZED][text/plain; charset=UTF-8][0] sent response to [Netty4HttpChannel{localAddress=/10.0.5.8:9200, remoteAddress=/10.0.3.4:32528}] success [true]
[2021-09-15T09:08:24,866][TRACE][c.a.o.s.a.BackendRegistry] [es3.logs.example.com] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'
```
Kibana Server Logs
```text
Sep 14 15:33:17 kb1.logs.example.com kibana[493]: {"type":"log","@timestamp":"2021-09-14T15:33:17Z","tags":["error","elasticsearch","data"],"pid":493,"message":"[ResponseError]: Response Error"}
Sep 14 15:33:17 kb1.logs.example.com kibana[493]: {"type":"log","@timestamp":"2021-09-14T15:33:17Z","tags":["error","http"],"pid":493,"message":"{ ResponseError: Response Error\n at IncomingMessage.response.on (/us
r/share/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:272:25)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/proc
ess/next_tick.js:63:19)\n name: 'ResponseError',\n meta:\n { body: '',\n statusCode: 401,\n headers:\n { 'x-opaque-id': '{SOME_UUID}',\n 'www-authenticate':\n 'X-S
ecurity-IdP realm=\"Open Distro Security\" location=\"https://subdomain.okta.com/app/xxx/yyy/sso/saml?SAMLRequest=some-base64-data\" requestId=\"ONELOGIN_{SOME_UUID}\"',\n 'content-type': 'text/plain; charset=UTF-8',\n 'content-length': '0' },\n
meta:\n { context: null,\n request: [Object],\n name: 'elasticsearch-js',\n connection: [Object],\n attempts: 0,\n aborted: false } },\n isBoom: true,\n isServer: false,\n d
ata: null,\n output:\n { statusCode: 401,\n payload:\n { statusCode: 401,\n error: 'Unauthorized',\n message: 'Response Error' },\n headers: {} },\n reformat: [Function],\n [Symbol(SavedOb
jectsClientErrorCode)]: 'SavedObjectsClient/notAuthorized' }"}
Sep 14 15:33:17 kb1.logs.example.com kibana[493]: {"type":"error","@timestamp":"2021-09-14T15:33:17Z","tags":[],"pid":493,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Inter
nal Server Error\n at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:69:19)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:177:34
)\n at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/","path":
"/","href":"/"},"message":"Internal Server Error"}
Sep 14 15:33:17 kb1.logs.example.com kibana[493]: {"type":"response","@timestamp":"2021-09-14T15:33:17Z","tags":[],"pid":493,"method":"get","statusCode":500,"req":{"url":"/","method":"get","headers":{"x-forwarded-for"
:"38.103.45.2","x-forwarded-proto":"https","x-forwarded-port":"443","host":"kb.logs.example.com","x-amzn-trace-id":"Root=some-id","sec-ch-ua":"\"Google Chrome\";v=\"93\", \" Not;A Brand\";v
=\"99\", \"Chromium\";v=\"93\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"macOS\"","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko
) Chrome/93.0.4577.63 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-m
ode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8,fr;q=0.7"},"remoteAddress":"10.0.1.36","userAgent":"Mozilla/5.0 (Macin
tosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36"},"res":{"statusCode":500,"responseTime":56,"contentLength":9},"message":"GET / 500 56ms - 9.0B"}
```
**Host/Environment (please complete the following information):**
- OS: Ubuntu 20.04
- ElasticSearch: OSS 7.10.2
- Kibana (`opendistroforelasticsearch-kibana`): 1.13.2
- OpenDistro versions:
```text
opendistro-alerting 1.13.1.0-1
opendistro-anomaly-detection 1.13.0.0-1
opendistro-asynchronous-search 1.13.0.1-1
opendistro-index-management 1.13.2.0-1
opendistro-job-scheduler 1.13.0.0-1
opendistro-knn 1.13.0.0-1
opendistro-knnlib 1.13.0.0
opendistro-performance-analyzer 1.13.0.0-1
opendistro-reports-scheduler 1.13.0.0-1
opendistro-security 1.13.1.0-1
opendistro-sql 1.13.2.0-1
opendistroforelasticsearch 1.13.2-1
```