Description
Before reporting an issue
- I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.
Area
No response
Describe the bug
When upgrading to a newer Keycloak(26.1.3 -> 26.2.3) in recreate and Cluster with more than 1 node, the DefaultMigrationManager runs migrate in each node. Specifically it runs
model.setStoredVersion(currentVersion.toString()); in more than 1 node.
This results in MigrationModelEntity table to have more than 1 row on the same version and the migration was ran more than once.
Also a custom theme was used. This affects the login(welcome/ui) pages UI to not load properly due to 404 in UI static resources fetching. In the requests with the 404, it can be seen that the Id from the MigrationModelEntity table is used so when trying to change it to the other id it works(if the request goes to the same node that created this row).
Each node is using a different Id from the MigrationModelEntity table in MigrationModelAdapter.latest
Maybe this is a missing configuration or something of this sort, but in previous upgrades(to different verisons) of same scenario it wasn't reproduced.
Version
26.2.3
Regression
- The issue is a regression
Expected behavior
MigrationModelEntity table to have a single row for each version.
When upgrading with Recreate + Cluster with more than 1 node, should run these migrations once and should be aware to handle this case of race.
Actual behavior
MigrationModelEntity table have more than 1 row for the version you upgraded to.
When upgrading with Recreate + Cluster with more than 1 node, runs these migrations more than once without logic of locking.
How to Reproduce?
- Set up Keycloak in version 26.1.3 or lower in Cluster with more than 1 node(can use 4 for easier reproduction)
- Start an upgrade of the environment to 26.2.3, Should be Recreate upgrade(add Debug logs for better visibility).
- Once the upgrade is completed, can be seen in
MigrationModelEntitytable that we have more than 1 row for 26.2.3 version
As this is some kind of race condition, can be a bit of flaky reproduction
Anything else?
Logs from 2 different pods:
local-keycloak-56bdb855f8-gc5jh local-keycloak 2025-05-19 13:00:41,596 INFO [org.keycloak.storage.datastore.DefaultMigrationManager] (main) Migrating older model to 26.2.0
local-keycloak-56bdb855f8-mw2r8 local-keycloak 2025-05-19 13:00:37,876 DEBUG [org.keycloak.quarkus.runtime.storage.database.jpa.QuarkusJpaConnectionProviderFactory] (main) Calling migrateModel
local-keycloak-56bdb855f8-mw2r8 local-keycloak 2025-05-19 13:00:38,580 INFO [org.keycloak.storage.datastore.DefaultMigrationManager] (main) Migrating older model to 26.2.0