Openid_auth: SelfRefreshingKeySet does not recover after JWKS fetch failures until node restart or security cache flush (OpenSearch 3.4.0)

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch: 3.4.0
Security plugin: bundled with 3.4.0
Auth backend: openid_auth (OIDC), IdP is Keycloak
JVM: OpenJDK 21
Host OS: Linux container image (SLES-based), running on Kubernetes
OpenSearch Dashboards: not involved
Browser: not applicable (service-to-service client using JWTs)

Describe the issue: we would like to understand how
the security plugin behaves while the JWKS endpoint is unreachable, and how it is expected to recover once it becomes reachable again. The transport failure itself is environmental and we are handling it separately.

Setup: openid_auth as the first authc domain, clientcert configured as a lower-priority fallback. Service-to-service clients authenticate with JWTs from the IdP. Two coordinating nodes.

What we observed on node 1, over roughly 28 minutes:

  • SelfRefreshingKeySet logged “Performing refresh N” with N climbing to 572, every
    attempt failing to fetch the discovery document
  • incoming JWTs were rejected throughout with “AuthenticatorUnavailableException: Too many unknown kids recently: 1373”
  • requests fell through to the clientcert domain, so callers were silently
    authenticated as a different, much less privileged identity
  • the sequence ended with the counter resetting to 1 and “KeySetProvider finished” appearing for the first and only time in the whole log, shortly after a security configuration reload. We cannot tell from the logs whether the reload caused the recovery or whether the underlying connectivity simply cleared at the same moment.

Node 2 ran continuously for six days. Its first “Performing refresh 1” appears two days into its lifetime, fails the same way, reaches “Performing refresh 55” about three hours later, and “KeySetProvider finished” never appears at all. Since a refresh seems to be driven by an incoming request with an unknown kid, we assume the attempts simply stopped when traffic stopped, rather than the node recovering.

Configuration:
config.yml (authc):
authc:
openid_auth_domain:
http_enabled: true
transport_enabled: true
order: 0
http_authenticator:
type: openid
challenge: false
config:
openid_connect_url: https://idp.example.com:8443/auth/realms/master/.well-known/openid-configuration
authentication_backend:
type: noop
jwt_auth_domain:
http_enabled: true
order: 1
clientcert_auth_domain:
http_enabled: true
order: 2
http_authenticator:
type: clientcert
authentication_backend:
type: noop

opensearch.yml
plugins.security.cache.ttl_minutes: 10

Relevant Logs or Screenshots:
The JWKS fetch fails at transport level. The specific transport cause is environmental and we are tracking it separately, so please treat it as given - our question is about what the plugin does from that point onwards. (We saw more than one underlying transport error in the same window; they surface identically through the security plugin, via the same call site below.)

{"severity":"warning","timestamp":"2026-10-11T22:21:59.059Z", "extra_data":{"class":"o.o.s.a.h.j.k.SelfRefreshingKeySet"}, "message":"KeySetProvider threw error
org.opensearch.security.auth.http.jwt.keybyoidc.AuthenticatorUnavailableException:
Error while getting https://idp.example.com:8443/auth/realms/master/.well-known/openid-configuration:
java.net.SocketException: Broken pipe at org.opensearch.security.auth.http.jwt.keybyoidc.KeySetRetriever.getJwksUri(KeySetRetriever.java:245) at org.opensearch.security.auth.http.jwt.keybyoidc.KeySetRetriever.get(KeySetRetriever.java:108)
at org.opensearch.security.auth.http.jwt.keybyoidc.SelfRefreshingKeySet$1.run(SelfRefreshingKeySet.java:213) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: java.net.SocketException: Broken pipe at java.base/sun.nio.ch.SocketDispatcher.write0(Native Method) ..."}


{"severity":"info","timestamp":"2026-10-09T00:05:12.012Z", "extra_data":{"class":"o.o.s.a.h.j.AbstractHTTPJwtAuthenticator"},
"message":"org.opensearch.security.auth.http.jwt.keybyoidc.AuthenticatorUnavailableException:Too many unknown kids recently: 1373"}

Node-1
2026-10-09T00:05:17.017Z   Performing refresh 492     (log starts here, already failing)
2026-10-09T00:33:03.003Z   Performing refresh 572
2026-10-09T00:35:00.000Z   KeySetProvider finished    (only occurrence in the whole log)
2026-10-09T00:35:00.000Z   Performing refresh 1       (counter restarts at 1)

Node-2
2026-10-11T22:21:59.059Z   Performing refresh 1       (first refresh on this node)
2026-10-11T22:21:59.059Z   KeySetProvider threw error ... SocketException: Broken pipe
2026-10-12T01:15:08.008Z   Performing refresh 55

Counts across both nodes: 7671 x “Too many unknown kids”, 1 x “KeySetProvider finished”

Questions/Clarifications:

  1. Is there any backoff or rate limiting on failed JWKS fetches? 572 attempts in 28
    minutes on one node seemed high to us, and we would like to know whether that is
    expected or whether it indicates something misconfigured on our side.

  2. What exactly does “Too many unknown kids recently: N” indicate, and does reaching
    that threshold change the refresh behaviour - for example suppressing further
    fetch attempts for a period?

  3. Once the IdP is reachable again, is SelfRefreshingKeySet expected to recover on
    its own, or does something have to reinitialise it? Is a refresh only triggered by
    an incoming request carrying an unknown kid, or is there also a background
    schedule?

  4. Does plugins.security.cache.ttl_minutes have any bearing on the OIDC key set, or
    does it only govern the authentication/authorization result cache? We would like
    to confirm whether cache expiry can refresh signing keys at all.

  5. Is the silent fallthrough to a lower-priority authc domain when the primary domain
    is unavailable the intended behaviour? From a caller’s point of view the request
    succeeds but with reduced privileges, which was harder to diagnose than an
    outright 503 would have been.

    //BR
    Chiranjeevi

1 Like