Description
Before reporting an issue
- I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.
Area
infinispan
Describe the bug
Hello,
I am working on a standalone Keycloak node in production mode (run with "start" and not "start-dev"). Since upgrading from version 23.0.7 (JDK 17) to 26.0.8 (JDK 21) then to 26.1.4 (JDK 21), I am having thread accumulation issues.
I do indeed see a thread accumulation with this stack trace:
"at jdk.internal.misc.Unsafe.park(java.base@21.0.6/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@21.0.6/LockSupport.java:221)
at java.util.concurrent.CompletableFuture$Signaller.block(java.base@21.0.6/CompletableFuture.java:1864)
at org.keycloak.models.sessions.infinispan.changes.JpaChangesPerformer.applyChanges(JpaChangesPerformer.java:109)
at org.keycloak.models.sessions.infinispan.changes.PersistentSessionsChangelogBasedTransaction$$Lambda/0x00007f7798135aa8.accept(Unknown Source)
at java.lang.Iterable.forEach(java.base@21.0.6/Iterable.java:75)
at org.keycloak.models.sessions.infinispan.changes.PersistentSessionsChangelogBasedTransaction.commitImpl(PersistentSessionsChangelogBasedTransaction.java:222)
at org.keycloak.models.AbstractKeycloakTransaction.commit(AbstractKeycloakTransaction.java:46)
at org.keycloak.services.DefaultKeycloakTransactionManager.lambda$commitWithTracing$0(DefaultKeycloakTransactionManager.java:169)
at org.keycloak.services.DefaultKeycloakTransactionManager$$Lambda/0x00007f7797f31b70.accept(Unknown Source)
at org.keycloak.quarkus.runtime.tracing.OTelTracingProvider.trace(OTelTracingProvider.java:117)
at org.keycloak.tracing.TracingProvider.trace(TracingProvider.java:174)
at org.keycloak.services.DefaultKeycloakTransactionManager.commitWithTracing(DefaultKeycloakTransactionManager.java:168)
at org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:146)
at org.keycloak.services.DefaultKeycloakSession.closeTransactionManager(DefaultKeycloakSession.java:396)
at org.keycloak.services.DefaultKeycloakSession.close(DefaultKeycloakSession.java:361)
at org.keycloak.models.KeycloakBeanProducer_ProducerMethod_getKeycloakSession_XoSEUTXOsE3bpqXlGMAykCiECUM_ClientProxy.close(Unknown Source)
at org.keycloak.quarkus.runtime.transaction.TransactionalSessionHandler.close(TransactionalSessionHandler.java:60)
at org.keycloak.quarkus.runtime.integration.jaxrs.CloseSessionFilter.closeSession(CloseSessionFilter.java:67)
at org.keycloak.quarkus.runtime.integration.jaxrs.CloseSessionFilter.filter(CloseSessionFilter.java:63)
at org.jboss.resteasy.reactive.server.handlers.ResourceResponseFilterHandler.handle(ResourceResponseFilterHandler.java:25)
at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:150)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:635)
at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.runWith(java.base@21.0.6/Thread.java:1596)
at java.lang.Thread.run(java.base@21.0.6/Thread.java:1583)"
It seems to be an Infinispan problem but I'm not sure. When using a standalone node, do I have to specify "cache=local" at startup from now on ?
I'm getting this issue with one of my environment but not all my standalone environments so I'm not able to reproduce it easily.
If it can help, I have metrics and telemetry features enabled, but not histograms metrics :
health-enabled=true
metrics-enabled=true
cache-metrics-histograms-enabled=false
http-metrics-histograms-enabled=false
tracing-enabled=true
tracing-endpoint=$PROM_SERVER
tracing-service-name=$SERIVCE_NAME
Version
26.1.4
Regression
- The issue is a regression
Expected behavior
Not having a thread accumulation
Actual behavior
JVM threads only increase at night and at some point I can log in with my password, but on the OTP login page I get a timeout. If I don't restart the service it becomes completely unavailable after a while.
How to Reproduce?
Standalone node started with "start" command without 'cache' configuration.
Anything else?
No response