Open
Description
Before reporting an issue
- I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.
Area
operator
Describe the bug
Hi, after recent update from version 26.0.7 to 26.2.4 (tried also 26.2.5) the operator started failing when keycloak manifest change.
The logs from operator:
Root cause:
1749020241203 2025-06-04 06:57:21,098 WARN [io.ver.cor.imp.BlockedThreadChecker] (vertx-blocked-thread-checker) Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2599 ms, time limit is 2000 ms: io.vertx.core.VertxException: Thread blocked
1749020241098
1749020241098 at java.base/java.lang.Thread.run(Thread.java:1583)
1749020241098 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
1749020241098 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
1749020241098 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
1749020241098 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
1749020241098 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
1749020241098 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
1749020241098 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)
1749020241098 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
1749020241098 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
1749020241098 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
1749020241098 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
1749020241098 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
1749020241098 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
1749020241098 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
1749020241098 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
1749020241098 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
1749020241098 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
1749020241098 at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
1749020241098 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1428)
1749020241098 at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1377)
1749020241098 at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1519)
1749020241098 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
1749020241097 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
1749020241097 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
1749020241097 at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
1749020241097 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
1749020241097 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
1749020241097 at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
1749020241097 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
1749020241097 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
1749020241097 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
1749020241097 at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153)
1749020241097 at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:159)
1749020241097 at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:799)
1749020241097 at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:837)
1749020241097 at io.vertx.core.http.impl.Http1xClientConnection.handleResponseEnd(Http1xClientConnection.java:965)
1749020241097 at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:307)
1749020241097 at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:327)
1749020241097 at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:734)
1749020241097 at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:157)
1749020241097 at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:279)
1749020241097 at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:424)
1749020241097 at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250)
1749020241097 at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:76)
1749020241097 at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:252)
1749020241097 at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270)
1749020241097 at io.fabric8.kubernetes.client.vertx.VertxHttpRequest.lambda$consumeBytes$1(VertxHttpRequest.java:120)
1749020241097 at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
1749020241097 at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
1749020241097 at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
1749020241097 at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
1749020241097 at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51)
1749020241097 at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
1749020241097 at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
1749020241097 at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
1749020241097 at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
1749020241097 at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:141)
1749020241097 at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
1749020241097 at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
1749020241097 at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
1749020241097 at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:551)
1749020241097 at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:257)
1749020241097 at com.fasterxml.jackson.databind.ObjectMapper.readerFor(ObjectMapper.java:4379)
1749020241097 at com.fasterxml.jackson.databind.ObjectMapper._newReader(ObjectMapper.java:809)
1749020241097 at com.fasterxml.jackson.databind.ObjectReader.<init>(ObjectReader.java:194)
1749020241097 at com.fasterxml.jackson.databind.ObjectReader._prefetchRootDeserializer(ObjectReader.java:2450)
1749020241097 at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:669)
1749020241097 at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:174)
1749020241097 at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:265)
1749020241097 at java.base/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:322)
1749020241097 at java.base/java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:153)
1749020241097 at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:990)
1749020241097 at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:754)
1749020241097 at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
1749020241097 at java.base/jdk.internal.misc.Unsafe.park(Native Method)
What helps is keycloak operator deployment full removal (sometimes a couple of times) and in some specific run the opertor works fine until next keycloak template diff. When this situation occur without manual intervention operator does not return to working state on his own.
We did not register similar behavior in previous versions of operator.
Version
26.2.4, 26.2.5
Regression
- The issue is a regression
Expected behavior
Stable operator working.
Actual behavior
Keycloak diff cause failing (constantly restarting) operator.
How to Reproduce?
Install latest operator version (ie. 26.2.4, 26.2.5), generate diff on keycloak template handled by operator.
Anything else?
No response