8000 JGroups: Switch to "per-destination" bundler for all TCP based stacks · Issue #39545 · keycloak/keycloak · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

JGroups: Switch to "per-destination" bundler for all TCP based stacks #39545

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ahus1 opened this issue May 7, 2025 · 4 comments
Open

JGroups: Switch to "per-destination" bundler for all TCP based stacks #39545

ahus1 opened this issue May 7, 2025 · 4 comments
Labels
kind/enhancement Categorizes a PR related to an enhancement status/triage team/sre
Milestone
26.3.0

Comments

@ahus1
Copy link
Contributor
ahus1 commented May 7, 2025

Description

The default bundler for the kubernetes stack as well all (?) other TCP based stacks is tq-bundler.

A per-destination bundler would avoid blocking other nodes when one node becomes unreachable.

Discussion

#39023 (comment)

Motivation

As seen in #39023 opening a TCP connection is a blocking activity. Similarly other operations on a TCP socket are blocking when the queue is full.

To avoid disruptions, use a bundler that creates one thread per destination. As the number of Keycloak nodes is usually well below 20, this shouldn't become a problem. Especially that we now also support virtual threads.

Details

No response
@ahus1 ahus1 added kind/enhancement Categorizes a PR related to an enhancement status/triage team/sre labels May 7, 2025
@ahus1 ahus1 added this to the 26.3.0 milestone May 7, 2025
@ahus1 ahus1 mentioned this issue May 7, 2025
Closed
2 tasks
@ahus1
Copy link
Contributor Author
ahus1 commented May 13, 2025

Note: per-destination is an experimental JGroups bundler: https://issues.redhat.com/browse/JGRP-2878

@ahus1
Copy link
Contributor Author
ahus1 commented May 13, 2025

When doing a rolling restart of pods, there is a moment when the Pod keycloak-1 left the cluster, and the current Pod is still trying to probe for the old destination.

Each of those probes will block the tq-bundler. The test below was run with the pd-bundler as that bundler will log the error messages, which the tq-bundler does not.

{"timestamp":"2025-05-07T18:46:09.78752888Z","sequence":9963,"loggerClassName":"org.infinispan.util.logging.Log_$logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"ISPN000078: Starting JGroups channel `ISPN` with stack `kubernetes-patched`","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:09.788847446Z","sequence":9964,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.JChannel","level":"INFO","message":"local_addr: fbb61ec2-2ec4-4f6c-a29f-1a7b064ae487, name: keycloak-2-17155","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:09.798488227Z","sequence":9965,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.FD_SOCK2","level":"INFO","message":"server listening on *:57800","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:09.926413972Z","sequence":9970,"loggerClassName":"org.infinispan.util.logging.Log_$logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"ISPN000094: Received new cluster view for channel ISPN: [keycloak-1-12651|13] (3) [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155]","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:09.929788623Z","sequence":9971,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.keycloak.infinispan.module.certificates.CertificateReloadManager","level":"INFO","message":"Reloading JGroups Certificate","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:09.972537738Z","sequence":9972,"loggerClassName":"org.infinispan.util.logging.Log_$logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"ISPN000079: Channel `ISPN` local address is `keycloak-2-17155`, physical addresses are `[10.129.2.20:7800]`","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.158217622Z","sequence":9973,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=org.infinispan.CONFIG] ISPN100002: Starting rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 54","threadName":"","threadId":106,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.176169837Z","sequence":9974,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=org.infinispan.CONFIG] ISPN100010: Finished rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], topology id 54","threadName":"non-blocking-thread--p2-t5","threadId":108,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.213735725Z","sequence":9975,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=actionTokens] ISPN100002: Starting rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 63","threadName":"","threadId":126,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.218016331Z","sequence":9976,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=actionTokens] ISPN100010: Finished rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], topology id 63","threadName":"non-blocking-thread--p2-t4","threadId":105,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.247752082Z","sequence":9977,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=authenticationSessions] ISPN100002: Starting rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 60","threadName":"","threadId":138,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.250453934Z","sequence":9978,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=authenticationSessions] ISPN100010: Finished rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], topology id 60","threadName":"","threadId":140,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.2955015Z","sequence":9979,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=clientSessions] ISPN100002: Starting rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 52","threadName":"","threadId":151,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.298057635Z","sequence":9980,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=clientSessions] ISPN100010: Finished rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], topology id 52","threadName":"","threadId":153,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.320216411Z","sequence":9981,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=loginFailures] ISPN100002: Starting rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 59","threadName":"","threadId":163,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.322795025Z","sequence":9982,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=loginFailures] ISPN100010: Finished rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], topology id 59","threadName":"","threadId":165,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.339094431Z","sequence":9983,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=offlineClientSessions] ISPN100002: Starting rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 54","threadName":"","threadId":175,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.341357844Z","sequence":9984,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=offlineClientSessions] ISPN100010: Finished rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], topology id 54","threadName":"","threadId":177,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.35601811Z","sequence":9985,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=offlineSessions] ISPN100002: Starting rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 53","threadName":"","threadId":187,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.35818616Z","sequence":9986,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=offlineSessions] ISPN100010: Finished rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], topology id 53","threadName":"","threadId":189,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.375062544Z","sequence":9987,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=sessions] ISPN100002: Starting rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 52","threadName":"","threadId":199,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.377290205Z","sequence":9988,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=sessions] ISPN100010: Finished rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], topology id 52","threadName":"","threadId":201,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.394144847Z","sequence":9989,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=work] ISPN100002: Starting rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 52","threadName":"","threadId":211,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.395465463Z","sequence":9990,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=work] ISPN100010: Finished rebalance with members [keycloak-1-12651, keycloak-0-30018, keycloak-2-17155], topology id 52","threadName":"non-blocking-thread--p2-t8","threadId":116,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:10.463607296Z","sequence":9991,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory","level":"INFO","message":"Node name: keycloak-2-17155, Site name: null","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.218237872Z","sequence":9994,"loggerClassName":"org.jboss.logging.Logger","loggerName":"io.quarkus","level":"INFO","message":"Keycloak 999.0.0-SNAPSHOT on JVM (powered by Quarkus 3.20.0) started in 6.262s. Listening on: https://0.0.0.0:8443. Management interface listening on https://0.0.0.0:9000.","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.218755415Z","sequence":9995,"loggerClassName":"org.jboss.logging.Logger","loggerName":"io.quarkus","level":"INFO","message":"Profile prod activated. ","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.218936021Z","sequence":9996,"loggerClassName":"org.jboss.logging.Logger","loggerName":"io.quarkus","level":"INFO","message":"Installed features: [agroal, cdi, hibernate-orm, jdbc-postgresql, keycloak, micrometer, narayana-jta, opentelemetry, reactive-routes, rest, rest-jackson, smallrye-context-propagation, smallrye-health, vertx]","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.596746545Z","sequence":10157,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=authenticationSessions] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 65","threadName":"","threadId":243,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.598026582Z","sequence":10158,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=authenticationSessions] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155], topology id 65","threadName":"non-blocking-thread--p2-t7","threadId":113,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.603922196Z","sequence":10159,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=sessions] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 57","threadName":"","threadId":251,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.605042999Z","sequence":10160,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=sessions] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155], topology id 57","threadName":"non-blocking-thread--p2-t10","threadId":120,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.6065429Z","sequence":10161,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=clientSessions] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 57","threadName":"","threadId":254,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.607400012Z","sequence":10162,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=clientSessions] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155], topology id 57","threadName":"non-blocking-thread--p2-t5","threadId":108,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.609186365Z","sequence":10163,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=work] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 57","threadName":"","threadId":260,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.609958416Z","sequence":10164,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=work] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155], topology id 57","threadName":"non-blocking-thread--p2-t9","threadId":118,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.611927437Z","sequence":10165,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=loginFailures] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 64","threadName":"","threadId":267,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.612951538Z","sequence":10166,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=loginFailures] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155], topology id 64","threadName":"non-blocking-thread--p2-t5","threadId":108,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.615289505Z","sequence":10167,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=offlineClientSessions] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 59","threadName":"","threadId":279,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.616100282Z","sequence":10168,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=offlineClientSessions] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155], topology id 59","threadName":"non-blocking-thread--p2-t4","threadId":105,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.617762205Z","sequence":10169,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=org.infinispan.CONFIG] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 59","threadName":"","threadId":288,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.618589718Z","sequence":10170,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=org.infinispan.CONFIG] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155], topology id 59","threadName":"","threadId":288,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.620994939Z","sequence":10171,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=offlineSessions] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 58","threadName":"","threadId":296,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.621786205Z","sequence":10172,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=offlineSessions] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155], topology id 58","threadName":"non-blocking-thread--p2-t10","threadId":120,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.623361463Z","sequence":10173,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=actionTokens] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 68","threadName":"","threadId":303,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.624199137Z","sequence":10174,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=actionTokens] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155], topology id 68","threadName":"","threadId":303,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:11.63423949Z","sequence":10175,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.ConnectException: Connection refused","threadName":"pd-bundler-7,keycloak-2-17155","threadId":79,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:12.003580126Z","sequence":10176,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.ConnectException: Connection refused","threadName":"pd-bundler-7,keycloak-2-17155","threadId":79,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:12.503747257Z","sequence":10179,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-7,keycloak-2-17155","threadId":79,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:13.406576572Z","sequence":10180,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-7,keycloak-2-17155","threadId":79,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:14.00859928Z","sequence":10181,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-7,keycloak-2-17155","threadId":79,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:14.610478122Z","sequence":10182,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-7,keycloak-2-17155","threadId":79,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:15.212366602Z","sequence":10183,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-7,keycloak-2-17155","threadId":79,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:15.814156603Z","sequence":10184,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-7,keycloak-2-17155","threadId":79,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:16.415969622Z","sequence":10185,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-7,keycloak-2-17155","threadId":79,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:16.416254741Z","sequence":10186,"loggerClassName":"org.infinispan.util.logging.Log_$logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"ISPN000094: Received new cluster view for channel ISPN: [keycloak-0-30018|14] (2) [keycloak-0-30018, keycloak-2-17155]","threadName":"","threadId":365,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:16.416616408Z","sequence":10187,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.keycloak.infinispan.module.certificates.CertificateReloadManager","level":"INFO","message":"Reloading JGroups Certificate","threadName":"","threadId":365,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:16.422952916Z","sequence":10188,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"ISPN100001: Node keycloak-1-12651 left the cluster","threadName":"","threadId":365,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:16.423282582Z","sequence":10189,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"ISPN100001: Node keycloak-1-12651 lef
8000
t the cluster","threadName":"","threadId":365,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:16.444566832Z","sequence":10190,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=offlineSessions] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 60","threadName":"","threadId":386,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:16.445272528Z","sequence":10191,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=offlineSessions] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155], topology id 60","threadName":"non-blocking-thread--p2-t7","threadId":113,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:16.447768406Z","sequence":10192,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=actionTokens] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155], phase READ_OLD_WRITE_ALL, topology id 70","threadName":"","threadId":395,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:16.448420059Z","sequence":10193,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=actionTokens] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155], topology id 70","threadName":"non-blocking-thread--p2-t8","threadId":116,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:16.716746869Z","sequence":10194,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-12,keycloak-2-17155","threadId":369,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:17.017456987Z","sequence":10195,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-7,keycloak-2-17155","threadId":79,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:17.318150063Z","sequence":10196,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-12,keycloak-2-17155","threadId":369,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:18.22090364Z","sequence":10197,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-12,keycloak-2-17155","threadId":369,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:18.822679183Z","sequence":10198,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-12,keycloak-2-17155","threadId":369,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:19.423385233Z","sequence":10199,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-12,keycloak-2-17155","threadId":369,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:20.025203103Z","sequence":10200,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-12,keycloak-2-17155","threadId":369,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:20.626972711Z","sequence":10201,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-12,keycloak-2-17155","threadId":369,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:21.228644449Z","sequence":10202,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to keycloak-1-12651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-12,keycloak-2-17155","threadId":369,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:30.937028969Z","sequence":10208,"loggerClassName":"org.infinispan.util.logging.Log_$logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"ISPN000094: Received new cluster view for channel ISPN: [keycloak-0-30018|15] (3) [keycloak-0-30018, keycloak-2-17155, 55a3a310-ba6a-46bd-889f-46272b95fadf]","threadName":"","threadId":561,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:30.937382889Z","sequence":10209,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.keycloak.infinispan.module.certificates.CertificateReloadManager","level":"INFO","message":"Reloading JGroups Certificate","threadName":"","threadId":561,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:30.941450912Z","sequence":10211,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"ISPN100000: Node keycloak-1-62279 joined the cluster","threadName":"","threadId":561,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:30.941776816Z","sequence":10212,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"ISPN100000: Node keycloak-1-62279 joined the cluster","threadName":"","threadId":561,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:31.191257658Z","sequence":10213,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=org.infinispan.CONFIG] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155, keycloak-1-62279], phase READ_OLD_WRITE_ALL, topology id 64","threadName":"","threadId":575,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:31.192258831Z","sequence":10214,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=org.infinispan.CONFIG] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155, keycloak-1-62279], topology id 64","threadName":"non-blocking-thread--p2-t4","threadId":105,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:31.238073986Z","sequence":10215,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-2-17155: failed sending message to 10.128.2.57:7800: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-14,keycloak-2-17155","threadId":566,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:31.247647638Z","sequence":10216,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=actionTokens] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155, keycloak-1-62279], phase READ_OLD_WRITE_ALL, topology id 74","threadName":"","threadId":586,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:31.248616707Z","sequence":10217,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=actionTokens] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155, keycloak-1-62279], topology id 74","threadName":"non-blocking-thread--p2-t2","threadId":96,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:31.291490916Z","sequence":10218,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=authenticationSessions] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155, keycloak-1-62279], phase READ_OLD_WRITE_ALL, topology id 70","threadName":"","threadId":593,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:31.292385905Z","sequence":10219,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=authenticationSessions] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155, keycloak-1-62279], topology id 70","threadName":"non-blocking-thread--p2-t4","threadId":105,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:31.340536414Z","sequence":10220,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=clientSessions] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155, keycloak-1-62279], phase READ_OLD_WRITE_ALL, topology id 62","threadName":"","threadId":600,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:31.341444007Z","sequence":10221,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=clientSessions] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155, keycloak-1-62279], topology id 62","threadName":"non-blocking-thread--p2-t2","threadId":96,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:31.365871741Z","sequence":10222,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=loginFailures] ISPN100002: Starting rebalance with members [keycloak-0-30018, keycloak-2-17155, keycloak-1-62279], phase READ_OLD_WRITE_ALL, topology id 69","threadName":"","threadId":611,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-07T18:46:31.366667656Z","sequence":10223,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=loginFailures] ISPN100010: Finished rebalance with members [keycloak-0-30018, keycloak-2-17155, keycloak-1-62279], topology id 69","threadName":"non-blocking-thread--p2-t10","threadId":120,"mdc":{},"ndc":"","hostName":"keycloak-2","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}

@ahus1
Copy link
Contributor Author
ahus1 commented May 13, 2025

Next steps:

  • Wait for Bela to respond on JIRA issue
  • KC Performance test single site with default tq-bundler and per-destination
  • KC Pod rolling restart / chaos test single site test to see if there are differences in latency spikes on the surviving Pods
  • Consider a bundler change in KC 26.3+, but keep default bundler in ISPN (tbd)
  • Ask ISPN to run a performance test with the per-destination bundler

@ahus1
Copy link
Contributor Author
ahus1 commented May 15, 2025

I tested with jdbc-ping on Kubernetes and a graceful start of a new + per-destination bundler. There is still a log of connection errors after a node is gracefully leaving. Tested with KC 26.3 nightly build.

{"timestamp":"2025-05-15T07:48:21.234652016Z","sequence":10358,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"[Context=offlineSessions] ISPN100010: Finished rebalance with members [keycloak-0-57063, keycloak-2-42409], topology id 14","threadName":"","threadId":1283,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:21.244064944Z","sequence":10359,"loggerClassName":"org.infinispan.util.logging.Log_$logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"ISPN000094: Received new cluster view for channel ISPN: [keycloak-0-57063|3] (2) [keycloak-0-57063, keycloak-2-42409]","threadName":"","threadId":1284,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:21.244355323Z","sequence":10360,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"[Context=actionTokens] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 12","threadName":"","threadId":1285,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:21.244668017Z","sequence":10361,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-0-57063: failed sending message to keycloak-1-59651: java.net.ConnectException: Connection refused","threadName":"pd-bundler-12,keycloak-0-57063","threadId":1294,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:21.24518746Z","sequence":10362,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"ISPN100001: Node keycloak-1-59651 left the cluster","threadName":"","threadId":1284,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:21.24534549Z","sequence":10363,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"ISPN100001: Node keycloak-1-59651 left the cluster","threadName":"","threadId":1284,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:21.245606494Z","sequence":10364,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-0-57063: failed sending message to keycloak-1-59651: java.net.ConnectException: Connection refused","threadName":"pd-bundler-12,keycloak-0-57063","threadId":1294,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:21.246427267Z","sequence":10365,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"[Context=actionTokens] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 13","threadName":"","threadId":1296,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:21.248636439Z","sequence":10366,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"[Context=actionTokens] ISPN100010: Finished rebalance with members [keycloak-0-57063, keycloak-2-42409], topology id 14","threadName":"","threadId":1297,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:21.330955309Z","sequence":10369,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-0-57063: failed sending message to keycloak-1-59651: java.net.ConnectException: Connection refused","threadName":"pd-bundler-12,keycloak-0-57063","threadId":1294,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:21.531217314Z","sequence":10370,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-0-57063: failed sending message to keycloak-1-59651: java.net.ConnectException: Connection refused","threadName":"pd-bundler-12,keycloak-0-57063","threadId":1294,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:21.731596943Z","sequence":10371,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-0-57063: failed sending message to keycloak-1-59651: java.net.ConnectException: Connection refused","threadName":"pd-bundler-12,keycloak-0-57063","threadId":1294,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:21.931748244Z","sequence":10372,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-0-57063: failed sending message to keycloak-1-59651: java.net.ConnectException: Connection refused","threadName":"pd-bundler-12,keycloak-0-57063","threadId":1294,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:22.131908513Z","sequence":10373,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-0-57063: failed sending message to keycloak-1-59651: java.net.ConnectException: Connection refused","threadName":"pd-bundler-12,keycloak-0-57063","threadId":1294,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:22.632251022Z","sequence":10374,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-0-57063: failed sending message to keycloak-1-59651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-12,keycloak-0-57063","threadId":1294,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:22.933247959Z","sequence":10375,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-0-57063: failed sending message to keycloak-1-59651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-12,keycloak-0-57063","threadId":1294,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:23.535046731Z","sequence":10376,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-0-57063: failed sending message to keycloak-1-59651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-12,keycloak-0-57063","threadId":1294,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:24.136881485Z","sequence":10377,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-0-57063: failed sending message to keycloak-1-59651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-12,keycloak-0-57063","threadId":1294,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:24.738683017Z","sequence":10378,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-0-57063: failed sending message to keycloak-1-59651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-12,keycloak-0-57063","threadId":1294,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:25.34060336Z","sequence":10379,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-0-57063: failed sending message to keycloak-1-59651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-12,keycloak-0-57063","threadId":1294,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:25.942361545Z","sequence":10380,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"ERROR","message":"keycloak-0-57063: failed sending message to keycloak-1-59651: java.net.SocketTimeoutException: Connect timed out","threadName":"pd-bundler-12,keycloak-0-57063","threadId":1294,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:36.5986342Z","sequence":10385,"loggerClassName":"org.infinispan.util.logging.Log_$logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"ISPN000094: Received new cluster view for channel ISPN: [keycloak-0-57063|4] (3) [keycloak-0-57063, keycloak-2-42409, 0c863d78-5218-4361-9032-708a31a08c15]","threadName":"","threadId":1466,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:36.598974812Z","sequence":10386,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"ISPN100000: Node 0c863d78-5218-4361-9032-708a31a08c15 joined the cluster","threadName":"","threadId":1466,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:36.599137003Z","sequence":10387,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"ISPN100000: Node 0c863d78-5218-4361-9032-708a31a08c15 joined the cluster","threadName":"","threadId":1466,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:36.600312653Z","sequence":10388,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.jgroups.protocols.TCP","level":"WARN","message":"JGRP000032: keycloak-0-57063: no physical address for 0c863d78-5218-4361-9032-708a31a08c15, dropping message","threadName":"pd-bundler-14,keycloak-0-57063","threadId":1467,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:37.229814464Z","sequence":10390,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.CLUSTER","level":"INFO","message":"[Context=org.infinispan.CONFIG] ISPN100002: Starting rebalance with members [keycloak-0-57063, keycloak-2-42409, keycloak-1-44426], phase READ_OLD_WRITE_ALL, topology id 15","threadName":"","threadId":1488,"mdc":{},"ndc":"","hostName":"keycloak-0","processName":"/usr/lib/jvm/java-21-openjdk-21.0.7.0.6-1.el9.aarch64/bin/java","processId":1}
{"timestamp":"2025-05-15T07:48:37.230296213Z","sequence":10391,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.infinispan.LIFECYCLE","level":"INFO","message":"[Context=org.infinispan.CONFIG] ISPN100002: Starting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/enhancement Categorizes a PR related to an enhancement status/triage team/sre
Projects
None yet
Milestone
26.3.0
Development

No branches or pull requests
1 participant
@ahus1
0