8000 Prevent a log-and-throw situation where errors appear in the log that are handled by ahus1 · Pull Request #39344 · keycloak/keycloak · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Prevent a log-and-throw situation where errors appear in the log that are handled #39344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

ahus1
Copy link
Contributor
@ahus1 ahus1 commented Apr 29, 2025

Closes #39173

Example log messages that are suppressed by this filtering:

2025-04-29 20:57:39,033 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker) ERROR: duplicate key value violates unique constraint "constraint_offl_cl_ses_pk3"
  Detail: Key (user_session_id, client_id, client_storage_provider, external_client_id, offline_flag)=(f818bfcc-7ec4-4a77-815c-5709e56737d7, 5cb62d8a-49ac-41d2-b554-d3b6df03055e, local, local, 0) already exists.
2025-04-29 20:57:39,045 WARN  [com.arjuna.ats.arjuna] (org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker) ARJUNA012125: TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple< 0:ffffc0a80156:96af:681120cd:43, org.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization@12b005ef >: org.hibernate.exception.ConstraintViolationException: could not execute batch [Batch entry 0 /* insert for org.keycloak.models.jpa.session.PersistentClientSessionEntity */insert into OFFLINE_CLIENT_SESSION (DATA,TIMESTAMP,VERSION,CLIENT_ID,CLIENT_STORAGE_PROVIDER,EXTERNAL_CLIENT_ID,OFFLINE_FLAG,USER_SESSION_ID) values (('{"authMethod":"openid-connect","redirectUri":"http://localhost:8080/admin/master/console/#/master/clients/5cb62d8a-49ac-41d2-b554-d3b6df03055e/sessions","notes":{"clientId":"5cb62d8a-49ac-41d2-b554-d3b6df03055e","iss":"http://localhost:8080/realms/master","startedAt":"1745952988","response_type":"code","level-of-authentication":"-1","code_challenge_method":"S256","nonce":"a660c5bc-e1ca-4275-95d5-6a9c9f4f36d8","response_mode":"query","scope":"openid","SSO_AUTH":"true","userSessionStartedAt":"1745951036","redirect_uri":"http://localhost:8080/admin/master/console/#/master/clients/5cb62d8a-49ac-41d2-b554-d3b6df03055e/sessions","state":"714e2aaa-e521-47ff-89e6-eb4a2a3a4df6","prompt":"none","code_challenge":"Ssevc0VDMW5S_YZDFWqQmSNyxv0sy-2lsNNj294uwaY"}}'),('1745952988'::int4),('0'::int4),('5cb62d8a-49ac-41d2-b554-d3b6df03055e'),('local'),('local'),('0'),('f818bfcc-7ec4-4a77-815c-5709e56737d7')) was aborted: ERROR: duplicate key value violates unique constraint "constraint_offl_cl_ses_pk3"
  Detail: Key (user_session_id, client_id, client_storage_provider, external_client_id, offline_flag)=(f818bfcc-7ec4-4a77-815c-5709e56737d7, 5cb62d8a-49ac-41d2-b554-d3b6df03055e, local, local, 0) already exists.  Call getNextException to see other errors in the batch.] [/* insert for org.keycloak.models.jpa.session.PersistentClientSessionEntity */insert into OFFLINE_CLIENT_SESSION (DATA,TIMESTAMP,VERSION,CLIENT_ID,CLIENT_STORAGE_PROVIDER,EXTERNAL_CLIENT_ID,OFFLINE_FLAG,USER_SESSION_ID) values (?,?,?,?,?,?,?,?)]
        at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:97)
        at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:58)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:108)
        at org.hibernate.engine.jdbc.batch.internal.BatchImpl.lambda$performExecution$2(BatchImpl.java:293)
        at org.hibernate.engine.jdbc.mutation.internal.PreparedStatementGroupSingleTable.forEachStatement(PreparedStatementGroupSingleTable.java:67)
        at org.hibernate.engine.jdbc.batch.internal.BatchImpl.performExecution(BatchImpl.java:264)
        at org.hibernate.engine.jdbc.batch.internal.BatchImpl.execute(BatchImpl.java:242)
        at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.executeBatch(JdbcCoordinatorImpl.java:188)
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:674)
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:511)
        at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:414)
        at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:41)
        at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:127)
        at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1429)
        at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:491)
        at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:2354)
        at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:1978)
        at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:439)
        at org.hibernate.resource.transaction.backend.jta.internal.JtaTransactionCoordinatorImpl.beforeCompletion(JtaTransactionCoordinatorImpl.java:336)
        at org.hibernate.resource.transaction.backend.jta.internal.synchronization.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:47)
        at org.hibernate.resource.transaction.backend.jta.internal.synchronization.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:37)
        at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:52)
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:348)
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:66)
        at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:135)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1307)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:104)
        at io.quarkus.narayana.jta.runtime.NotifyingTransactionManager.commit(NotifyingTransactionManager.java:70)
        at org.keycloak.transaction.JtaTransactionWrapper.commit(JtaTransactionWrapper.java:102)
        at org.keycloak.services.DefaultKeycloakTransactionManager.lambda$commitWithTracing$0(DefaultKeycloakTransactionManager.java:169)
        at org.keycloak.tracing.NoopTracingProvider.trace(NoopTracingProvider.java:59)
        at org.keycloak.tracing.NoopTracingProvider.trace(NoopTracingProvider.java:69)
        at org.keycloak.services.DefaultKeycloakTransactionManager.commitWithTracing(DefaultKeycloakTransactionManager.java:168)
        at org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
        at org.keycloak.services.DefaultKeycloakSession.closeTransactionManager(DefaultKeycloakSession.java:392)
        at org.keycloak.services.DefaultKeycloakSession.close(DefaultKeycloakSession.java:357)
        at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:467)
        at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:340)
        at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:330)
        at org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker.lambda$process$5(PersistentSessionsWorker.java:116)
        at org.keycloak.common.util.Retry.executeWithBackoff(Retry.java:102)
        at org.keycloak.common.util.Retry.executeWithBackoff(Retry.java:93)
        at org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker.lambda$process$7(PersistentSessionsWorker.java:113)
        at org.keycloak.models.utils.KeycloakModelUtils.lambda$runJobInTransaction$1(KeycloakModelUtils.java:341)
        at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransactionWithResult(KeycloakModelUtils.java:460)
        at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:340)
        at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:330)
        at org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker.process(PersistentSessionsWorker.java:97)
        at org.keycloak.models.sessions.infinispan.changes.PersistentSessionsWorker$BatchWorker.run(PersistentSessionsWorker.java:76)
Caused by: java.sql.BatchUpdateException: Batch entry 0 /* insert for org.keycloak.models.jpa.session.PersistentClientSessionEntity */insert into OFFLINE_CLIENT_SESSION (DATA,TIMESTAMP,VERSION,CLIENT_ID,CLIENT_STORAGE_PROVIDER,EXTERNAL_CLIENT_ID,OFFLINE_FLAG,USER_SESSION_ID) values (('{"authMethod":"openid-connect","redirectUri":"http://localhost:8080/admin/master/console/#/master/clients/5cb62d8a-49ac-41d2-b554-d3b6df03055e/sessions","notes":{"clientId":"5cb62d8a-49ac-41d2-b554-d3b6df03055e","iss":"http://localhost:8080/realms/master","startedAt":"1745952988","response_type":"code","level-of-authentication":"-1","code_challenge_method":"S256","nonce":"a660c5bc-e1ca-4275-95d5-6a9c9f4f36d8","response_mode":"query","scope":"openid","SSO_AUTH":"true","userSessionStartedAt":"1745951036","redirect_uri":"http://localhost:8080/admin/master/console/#/master/clients/5cb62d8a-49ac-41d2-b554-d3b6df03055e/sessions","state":"714e2aaa-e521-47ff-89e6-eb4a2a3a4df6","prompt":"none","code_challenge":"Ssevc0VDMW5S_YZDFWqQmSNyxv0sy-2lsNNj294uwaY"}}'),('1745952988'::int4),('0'::int4),('5cb62d8a-49ac-41d2-b554-d3b6df03055e'),('local'),('local'),('0'),('f818bfcc-7ec4-4a77-815c-5709e56737d7')) was aborted: ERROR: duplicate key value violates unique constraint "constraint_offl_cl_ses_pk3"
  Detail: Key (user_session_id, client_id, client_storage_provider, external_client_id, offline_flag)=(f818bfcc-7ec4-4a77-815c-5709e56737d7, 5cb62d8a-49ac-41d2-b554-d3b6df03055e, local, local, 0) already exists.  Call getNextException to see other errors in the batch.
        at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:165)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2421)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:580)
        at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:889)
        at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:913)
        at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1739)
        at io.agroal.pool.wrapper.StatementWrapper.executeBatch(StatementWrapper.java:340)
        at org.hibernate.engine.jdbc.batch.internal.BatchImpl.lambda$performExecution$2(BatchImpl.java:279)
        ... 45 more
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "constraint_offl_cl_ses_pk3"
  Detail: Key (user_session_id, client_id, client_storage_provider, external_client_id, offline_flag)=(f818bfcc-7ec4-4a77-815c-5709e56737d7, 5cb62d8a-49ac-41d2-b554-d3b6df03055e, local, local, 0) already exists.
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2733)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2420)
        ... 51 more

@ahus1 ahus1 self-assigned this Apr 29, 2025
@ahus1
Copy link
Contributor Author
ahus1 commented Apr 29, 2025

@vmuzikar - this is using the Quarkus log filtering. IMHO this is the appropriate concept to use here, please object if you think otherwise.

@pedroigor - FYI, this is then preventing a log-and-throw scenario where something is logged as either an error or warning, and then later handled. This should lead to clean log files when there is actually nothing to report.

Copy link
@keycloak-github-bot keycloak-github-bot bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unreported flaky test detected, please review

@keycloak-github-bot
Copy link

Unreported flaky test detected

If the flaky tests below are affected by the changes, please review and update the changes accordingly. Otherwise, a maintainer should report the flaky tests prior to merging the PR.

org.keycloak.testsuite.broker.KcOidcBrokerTest#loginWithExistingUserWithBruteForceEnabled

Keycloak CI - Java Distribution IT (windows-latest - temurin - 24)

org.openqa.selenium.NoSuchElementException: 
Unable to locate element
For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors#no-such-element-exception
Build info: version: '4.28.1', revision: '73f5ad48a2'
System info: os.name: 'Windows Server 2022', os.arch: 'amd64', os.version: '10.0', java.version: '24.0.1'
...

Report flaky test

Copy link
Contributor
@vmuzikar vmuzikar left a comment

Choose a reason for hiding this comment

The reason will be di 8000 splayed to describe this comment to others. Learn more.

This approach looks fine to me, though it'd be good to also see the possible perf implications of introducing a log filter.

… are handled

Closes keycloak#39173

Signed-off-by: Alexander Schwartz <aschwart@redhat.com>
@ahus1 ahus1 force-pushed the is-39173-suppress-log-and-throw-DB-errors branch from b528bb6 to 323fda9 Compare April 30, 2025 07:58
@ahus1
Copy link
Contributor Author
ahus1 commented Apr 30, 2025

@vmuzikar - I updated the log filter to compare the log level warn first, which should be faster than comparing strings. The comparison is done only for log messages that are logged, not all log messages that are generated, so we should be fine here.

@ahus1 ahus1 marked this pull request as ready for review April 30, 2025 08:09
@ahus1 ahus1 requested review from a team as code owners April 30, 2025 08:09
@ahus1 ahus1 requested a review from mhajas April 30, 2025 08:09
@ahus1
Copy link
Contributor Author
ahus1 commented Apr 30, 2025

@mhajas - please review when you have the time. Anyone else who wants to review is fine to review it as well. Will merge it on the first approving review :D

@ahus1
Copy link
Contributor Author
ahus1 commented May 2, 2025

Steps to reproduce this locally, follow these 12 steps:

  • Build a custom Keycloak version with a longer persistent worker retry period. Change it from 10 seconds to 1000 seconds.
  • Spin up Keycloak with a PostgreSQL database as it is running by default in read-committed isolation mode
  • Login to the admin UI with persistent sessions enabled (this is the default)
  • Go to the database and change the client session's reference entry to a different user session/client session (just dummy values) and commit so that KC will later try to insert a new row
  • Restart Keycloak to clear all caches
  • Please a breakpoint after the create client session is read from the databaes
  • Refresh your browser, so KC attempts to recreate the client session. Debugging session will stop at the breakpoint from the previous step
  • Update the previously manipulated client session, change it to the original client and user session values to create a DB conflict
  • Continue from the breakpoint. KC will insert a row. A DB exception will be thrown upon commit, but KC will retry.
  • With this changes no log entry should appear
  • Breakpoint will stop here again, but upon continue it will update and not insert.
  • Browser will not show any error to the user

Copy link
Contributor
@mhajas mhajas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR @ahus1, I added one comment

# SqlExceptionHelper will log-and-throw error messages.
# As those messages might later be caught and handled, this is an antipattern so we prevent logging them
# https://hibernate.zulipchat.com/#narrow/channel/132096-hibernate-user/topic/Feature.20Request.3A.20Disable.20logging.20of.20SqlExceptionHelper.20for
quarkus.log.category."org.hibernate.engine.jdbc.spi.SqlExceptionHelper".level=off
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on these messages, are we sure we want to disable all Hibernate exceptions logging?

If there is a place where we incorrectly handle an exception it could save us a lot of time investigating it if we have the exception printed in the log.

Don't get me wrong I am all in with minimizing log-and-throw pattern but I like the idea to filter one specific exception that is causing confusion the users based on the report rather than disabling logging for all Hibernate exceptions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Keycloak, we have the KeycloakErrorHandler that is handling this: The exception is either presented to the user and then it is logged in debug mode, or it is logged as an error.

In a situation that you describe, we could still enable debug logging (at least for the class KeycloakErrorHandler).

As this has been raised several times when we write to the DB in the persistent session worker, I'd rather proceed and hide those logs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, ok. I forgot all of the exceptions would end up in the Handler. Sorry for confusion.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, ok. I forgot all of the exceptions would end up in the Handler.

There might still be corner cases, but most of them should show in the handler.

Copy link
Contributor
@mhajas mhajas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification @ahus1. LGTM

# SqlExceptionHelper will log-and-throw error messages.
# As those messages might later be caught and handled, this is an antipattern so we prevent logging them
# https://hibernate.zulipchat.com/#narrow/channel/132096-hibernate-user/topic/Feature.20Request.3A.20Disable.20logging.20of.20SqlExceptionHelper.20for
quarkus.log.category."org.hibernate.engine.jdbc.spi.SqlExceptionHelper".level=off
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, ok. I forgot all of the exceptions would end up in the Handler. Sorry for confusion.

@mhajas mhajas merged commit 88a74b7 into keycloak:main May 6, 2025
80 checks passed
ahus1 added a commit to ahus1/keycloak that referenced this pull request May 6, 2025
… are handled (keycloak#39344)

Closes keycloak#39173

Signed-off-by: Alexander Schwartz <aschwart@redhat.com>
shawkins pushed a commit to shawkins/keycloak that referenced this pull request May 7, 2025
… are handled (keycloak#39344)

Closes keycloak#39173

Signed-off-by: Alexander Schwartz <aschwart@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

duplicate key value violates unique constraint "constraint_offl_cl_ses_pk3"
3 participants
0