8000 Performance with multiple pods in kubernetes · Issue #39304 · keycloak/keycloak · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Performance with multiple pods in kubernetes #39304

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
1 of 2 tasks
chuckbutkus opened this issue Apr 28, 2025 · 7 comments
Closed
1 of 2 tasks

Performance with multiple pods in kubernetes #39304

chuckbutkus opened this issue Apr 28, 2025 · 7 comments
Assignees
@ahus1
Labels
area/core kind/bug Categorizes a PR related to a bug team/cloud-native team/sre

Comments

@chuckbutkus
Copy link
chuckbutkus commented Apr 28, 2025 •
edited
Loading

Before reporting an issue

  • I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.

Area

core

Describe the bug

We previously ran version 26.1.1 and had no performance issues. After upgrading to 26.1.5 we saw our app pause occasionally. After investigation we found it was Keycloak after the upgrade. We tried the latest version (26.2.1) and it has the same pauses, so we had to revert back to 26.1.1.

Investigating further it appears to be something with how multiple pods interact. When we reduce the number of pods to one, everything is fine on all versions.

We install Keycloak via the bitnami helm chart, so it deploys as a statefulset.

Version

26.2.1

Regression

  • The issue is a regression

Expected behavior

Accessing Keycloak to get token or user data should not occasionally pause for several seconds.

Actual behavior

Accessing Keycloak to get token or user data occasionally pauses for several seconds.

How to Reproduce?

Run Keycloak (any version from 26.1.5 onward) with multiple pods (not sure if it matters, but we install via the bitnami helm chart). Then run the following python program:
import time

from keycloak.keycloak_admin import KeycloakAdmin

keycloak_admin = KeycloakAdmin(
    server_url='<keycloak_url>',
    username='admin',
    password='<admin-password>',
    realm_name='master',
    client_id='admin-cli',
    verify=True,
)

while True:
    start_time = time.perf_counter()
    user = keycloak_admin.get_user('<admin_user_id>')
    end_time = time.perf_counter()
    execution_time = end_time - start_time
    print(f'Execution time: {execution_time} seconds')
    time.sleep(2)

Against our install, we will often see long pauses (see below). On version 26.1.1 or with only a single pod we don't see these pauses.

Execution time: 0.6937680629926035 seconds
Execution time: 0.11748872800671961 seconds
Execution time: 0.11581164199742489 seconds
Execution time: 0.1149626660044305 seconds
Execution time: 9.4486416340078 seconds
Execution time: 0.11207192699657753 seconds
Execution time: 0.13379409599292558 seconds
Execution time: 0.11867227400944103 seconds
Execution time: 0.12262174100033008 seconds
Execution time: 0.11106013999960851 seconds
Execution time: 0.11135531299805734 seconds
Execution time: 3.2200807740009623 seconds
Execution time: 0.12285714299650863 seconds
Execution time: 0.11307278099411633 seconds
Execution time: 0.10820726399833802 seconds
Execution time: 5.010735514995758 seconds
Execution time: 0.11182817300141323 seconds
Execution time: 0.11489482400065754 seconds
Execution time: 11.430347254005028 seconds
Execution time: 0.10880520999489818 seconds
Execution time: 0.11470832000486553 seconds
Execution time: 0.11534533600206487 seconds
Execution time: 0.11483593100274447 seconds
Execution time: 0.11594764499750454 seconds
Execution time: 0.1217402089969255 seconds
Execution time: 0.1130012740031816 seconds
Execution time: 0.14769148999766912 seconds
Execution time: 4.140627556000254 seconds
Execution time: 7.464632677001646 seconds
Execution time: 0.11068905799766071 seconds
Execution time: 0.12943190200894605 seconds
Execution time: 0.12480702198809013 seconds
Execution time: 0.11917021199769806 seconds
Execution time: 0.12869390299601946 seconds
Execution time: 4.736036393995164 seconds
Execution time: 5.4778034089977155 seconds
Execution time: 0.17988387199875433 seconds
Execution time: 0.10829535599623341 seconds
Execution time: 4.5177753859898075 seconds
Execution time: 0.1093141530000139 seconds
Execution time: 4.880280893994495 seconds
Execution time: 5.347838064000825 seconds

Anything else?

This seems related to this issue: #39023
@chuckbutkus chuckbutkus added kind/bug Categorizes a PR related to a bug status/triage labels Apr 28, 2025
@chuckbutkus chuckbutkus changed the title Performance with mutltiple pods in kubernetes Performance with multiple pods in kubernetes Apr 29, 2025
@shawkins
Copy link
Contributor
shawkins commented May 1, 2025

@chuckbutkus could you provide the show-config output and / or the full Pod yaml to make sure we're not missing anything in attempting to reproduce this?

@chuckbutkus
Copy link
Author
chuckbutkus commented May 2, 2025 •
edited
Loading

@chuckbutkus could you provide the show-config output and / or the full Pod yaml to make sure we're not missing anything in attempting to reproduce this?

keycloak@keycloak-0:/$ /opt/bitnami/keycloak/bin/kc.sh show-config
Appending additional Java properties to JAVA_OPTS
Current Mode: production
Current Configuration:
        kc.health-enabled =  false (keycloak.conf)
        kc.cache-stack =  kubernetes (keycloak.conf)
        kc.db-username =  keycloak_user (keycloak.conf)
        kc.spi-admin-realm =  master (ENV)
        kc.log-level-org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup =  WARN (classpath application.properties)
        kc.log-level-io.quarkus.config =  off (classpath application.properties)
        kc.hostname =  https://latest-keycloak-vers.auth.staging.all-hands.dev/ (keycloak.conf)
        kc.log-console-output =  default (keycloak.conf)
        kc.hostname-strict =  true (keycloak.conf)
        kc.http-management-relative-path =  / (Persisted)
        kc.https-port =  8443 (keycloak.conf)
        kc.metrics-enabled =  false (keycloak.conf)
        kc.proxy-forwarded-host =  forwarded (keycloak.conf)
        kc.bootstrap-admin-username =  tmpadmin (ENV)
        kc.db =  postgres (keycloak.conf)
        kc.db-password =  ******* (keycloak.conf)
        kc.proxy-allow-forwarded-header =  forwarded (keycloak.conf)
        kc.log-level-io.quarkus.hibernate.orm.deployment.HibernateOrmProcessor =  warn (classpath application.properties)
        kc.version =  26.2.1 (SysPropConfigSource)
        kc.db-url =  jdbc:postgresql://openhands-latest-keycloak-vers-postgresql:5432/keycloak?currentSchema=public (keycloak.conf)
        kc.http-relative-path =  / (keycloak.conf)
        kc.features =  token-exchange,admin-fine-grained-authz (ENV)
        kc.http-enabled =  true (keycloak.conf)
        kc.proxy-trusted-header-enabled =  forwarded (keycloak.conf)
        kc.proxy-headers =  forwarded (keycloak.conf)
        kc.log-level-org.jboss.resteasy.resteasy_jaxrs.i18n =  WARN (classpath application.properties)
        kc.log-level-io.quarkus.arc.processor.BeanArchives =  off (classpath application.properties)
        kc.cache =  ispn (keycloak.conf)
        kc.log-level-io.quarkus.deployment.steps.ReflectiveHierarchyStep =  error (classpath application.properties)
        kc.http-port =  8080 (keycloak.conf)
        kc.bootstrap-admin-password-file =  /opt/bitnami/keycloak/secrets/admin-password (ENV)
        kc.proxy-allow-x-forwarded-header =  forwarded (keycloak.conf)
        kc.log-level =  INFO (keycloak.conf)
        kc.provider.file.keycloak-metrics-spi-7.0.0.jar.last-modified =  1746205753992 (Persisted)
        kc.log-level-io.quarkus.arc.processor.IndexClassLookupUtils =  off (classpath application.properties)

@chuckbutkus
Copy link
Author
chuckbutkus commented May 2, 2025 •
edited
Loading

@chuckbutkus could you provide the show-config output and / or the full Pod yaml to make sure we're not missing anything in attempting to reproduce this?

Pod YAML:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    checksum/configmap-env-vars: ec85bbddc976f5d7e80f02d77116a655120d88f669e83c48a85baa97c02eb21a
    cni.projectcalico.org/containerID: c250c4377740c4ab1ecefc0a7e9641bd06686f8d6636e1d1ac337be30ba8a48c
    cni.projectcalico.org/podIP: 10.105.1.90/32
    cni.projectcalico.org/podIPs: 10.105.1.90/32
  creationTimestamp: "2025-05-02T17:09:05Z"
  generateName: keycloak-
  labels:
    app.kubernetes.io/app-version: 26.2.1
    app.kubernetes.io/component: keycloak
    app.kubernetes.io/instance: openhands-latest-keycloak-vers
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: keycloak
    app.kubernetes.io/version: 26.2.1
    apps.kubernetes.io/pod-index: "0"
    controller-revision-hash: keycloak-8558d7787
    helm.sh/chart: keycloak-24.6.1
    statefulset.kubernetes.io/pod-name: keycloak-0
  name: keycloak-0
  namespace: openhands-latest-keycloak-vers
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: keycloak
    uid: eb3c562a-6e3e-40de-9119-13ca3862da3f
  resourceVersion: "235205483"
  uid: cb8be52d-a854-4221-b21f-96e46be7f479
spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchLabels:
              app.kubernetes.io/instance: openhands-latest-keycloak-vers
              app.kubernetes.io/name: keycloak
          topologyKey: kubernetes.io/hostname
        weight: 1
  automountServiceAccountToken: true
  containers:
  - env:
    - name: KUBERNETES_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: BITNAMI_DEBUG
      value: "false"
    - name: KC_BOOTSTRAP_ADMIN_PASSWORD_FILE
      value: /opt/bitnami/keycloak/secrets/admin-password
    - name: KEYCLOAK_DATABASE_PASSWORD_FILE
      value: /opt/bitnami/keycloak/secrets/db-db-password
    - name: KEYCLOAK_DATABASE_PORT_FILE
      value: /opt/bitnami/keycloak/secrets/db-db-port
    - name: KEYCLOAK_DATABASE_USER_FILE
      value: /opt/bitnami/keycloak/secrets/db-db-user
    - name: KEYCLOAK_DATABASE_NAME_FILE
      value: /opt/bitnami/keycloak/secrets/db-db-name
    - name: KEYCLOAK_HTTP_RELATIVE_PATH
      value: /
    - name: KC_SPI_ADMIN_REALM
      value: master
    - name: KC_FEATURES
      value: token-exchange,admin-fine-grained-authz
    envFrom:
    - configMapRef:
        name: keycloak-env-vars
    image: docker.io/bitnami/keycloak:26.2.1-debian-12-r0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      initialDelaySeconds: 300
      periodSeconds: 1
      successThreshold: 1
      tcpSocket:
        port: http
      timeoutSeconds: 5
    name: keycloak
    ports:
    - containerPort: 8080
      name: http
      protocol: TCP
    - containerPort: 7800
      name: discovery
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /realms/master
        port: http
        scheme: HTTP
      initialDelaySeconds: 30
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: 750m
        ephemeral-storage: 2Gi
        memory: 768Mi
      requests:
        cpu: 500m
        ephemeral-storage: 50Mi
        memory: 512Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: true
      runAsGroup: 1001
      runAsNonRoot: true
      runAsUser: 1001
      seLinuxOptions: {}
      seccompProfile:
        type: RuntimeDefault
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /tmp
      name: empty-dir
      subPath: tmp-dir
    - mountPath: /bitnami/keycloak
      name: empty-dir
      subPath: app-volume-dir
    - mountPath: /opt/bitnami/keycloak/conf
      name: empty-dir
      subPath: app-conf-dir
    - mountPath: /opt/bitnami/keycloak/lib/quarkus
      name: empty-dir
      subPath: app-quarkus-dir
    - mountPath: /opt/bitnami/keycloak/data
      name: empty-dir
      subPath: app-data-dir
    - mountPath: /opt/bitnami/keycloak/providers
      name: empty-dir
      subPath: app-providers-dir
    - mountPath: /opt/bitnami/keycloak/themes
      name: empty-dir
      subPath: app-themes-dir
    - mountPath: /opt/bitnami/keycloak/secrets
      name: keycloak-secrets
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qwphg
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: keycloak-0
  initContainers:
  - args:
    - -ec
    - |
      . /opt/bitnami/scripts/liblog.sh

      info "Copying writable dirs to empty dir"
      # In order to not break the application functionality we need to make some
      # directories writable, so we need to copy it to an empty dir volume
      cp -r --preserve=mode /opt/bitnami/keycloak/lib/quarkus /emptydir/app-quarkus-dir
      cp -r --preserve=mode /opt/bitnami/keycloak/data /emptydir/app-data-dir
      cp -r --preserve=mode /opt/bitnami/keycloak/providers /emptydir/app-providers-dir
      cp -r --preserve=mode /opt/bitnami/keycloak/themes /emptydir/app-themes-dir
      info "Copy operation completed"
    command:
    - /bin/bash
    image: docker.io/bitnami/keycloak:26.2.1-debian-12-r0
    imagePullPolicy: IfNotPresent
    name: prepare-write-dirs
    resources:
      limits:
        cpu: 750m
        ephemeral-storage: 2Gi
        memory: 768Mi
      requests:
        cpu: 500m
        ephemeral-storage: 50Mi
        memory: 512Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: true
      runAsGroup: 1001
      runAsNonRoot: true
      runAsUser: 1001
      seLinuxOptions: {}
      seccompProfile:
        type: RuntimeDefault
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /emptydir
      name: empty-dir
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qwphg
      readOnly: true
  - command:
    - sh
    - -c
    - |
      echo "Waiting for Postgres $DB_HOST to be available..."
      for i in {1..60}; do
        if psql -h $DB_HOST -p $DB_PORT -U postgres -d postgres -c "SELECT 1;" > /dev/null 2>&1; then
          echo "Postgres is up!"

          echo "Creating the database $DB_NAME if it doesn't exist..."
          # Create the database if it doesn't exist
          psql -h $DB_HOST -p $DB_PORT -U postgres -d postgres -tc "SELECT 1 FROM pg_database WHERE datname='$DB_NAME'" | grep -q 1 || \
          psql -h $DB_HOST -p $DB_PORT -U postgres -d postgres -c "CREATE DATABASE $DB_NAME;"

          echo "Creating the user $DB_USER if it doesn't exist..."
          psql -h $DB_HOST -p $DB_PORT -U postgres -d $DB_NAME -tc "SELECT 1 FROM pg_roles WHERE rolname='$DB_USER'" | grep -q 1 || \
          (psql -h $DB_HOST -p $DB_PORT -U postgres -d $DB_NAME -c "CREATE USER $DB_USER WITH PASSWORD '$DB_PASSWORD'; GRANT ALL PRIVILEGES ON DATABASE $DB_NAME TO $DB_USER; GRANT USAGE ON SCHEMA public TO $DB_USER; GRANT CREATE ON SCHEMA public TO $DB_USER; ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO $DB_USER;")

          exit 0
        fi
        echo "Waiting for Postgres..."
        sleep 10
      done
      echo "Postgres did not become available in time."
      exit 1
    env:
    - name: PGPASSWORD
      valueFrom:
        secretKeyRef:
          key: password
          name: postgres-password
    - name: DB_HOST
      value: openhands-latest-keycloak-vers-postgresql
    - name: DB_NAME
      valueFrom:
        secretKeyRef:
          key: db-name
          name: keycloak-db-secret
    - name: DB_PORT
      valueFrom:
        secretKeyRef:
          key: db-port
          name: keycloak-db-secret
    - name: DB_USER
      valueFrom:
        secretKeyRef:
          key: db-user
          name: keycloak-db-secret
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          key: db-password
          name: keycloak-db-secret
    image: postgres:14
    imagePullPolicy: IfNotPresent
    name: create-user
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qwphg
      readOnly: true
  nodeName: gke-staging-core-applicatio-primary-0-f75ac094-qygl
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1001
    fsGroupChangePolicy: Always
  serviceAccount: keycloak-sa
  serviceAccountName: keycloak-sa
  subdomain: keycloak-headless
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: empty-dir
  - name: keycloak-secrets
    projected:
      defaultMode: 420
      sources:
      - secret:
          name: keycloak-admin
      - secret:
          items:
          - key: db-password
            path: db-db-password
          - key: db-port
            path: db-db-port
          - key: db-user
            path: db-db-user
          - key: db-name
            path: db-db-name
          name: keycloak-db-secret
  - name: kube-api-access-qwphg
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2025-05-02T17:09:14Z"
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
    lastTransitionTime: "2025-05-02T17:09:29Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2025-05-02T17:10:46Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2025-05-02T17:10:46Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2025-05-02T17:09:05Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://6061469fdb2e36e6d0bd9834747c395ee7014b2c1366650fc55b686873fb5c7c
    image: docker.io/bitnami/keycloak:26.2.1-debian-12-r0
    imageID: docker.io/bitnami/keycloak@sha256:ab8978368be53cee475da4d798c7712c59cdae5eff5e636e679e0c8076d10364
    lastState: {}
    name: keycloak
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2025-05-02T17:09:29Z"
  hostIP: 10.104.3.224
  hostIPs:
  - ip: 10.104.3.224
  initContainerStatuses:
  - containerID: containerd://36c5cbdadc0a1c78d1104e5d33b2ddd7fcb3bde12dbb6e9d116816ceae23fd30
    image: docker.io/bitnami/keycloak:26.2.1-debian-12-r0
    imageID: docker.io/bitnami/keycloak@sha256:ab8978368be53cee475da4d798c7712c59cdae5eff5e636e679e0c8076d10364
    lastState: {}
    name: prepare-write-dirs
    ready: true
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://36c5cbdadc0a1c78d1104e5d33b2ddd7fcb3bde12dbb6e9d116816ceae23fd30
        exitCode: 0
       
8000
 finishedAt: "2025-05-02T17:09:13Z"
        reason: Completed
        startedAt: "2025-05-02T17:09:13Z"
  - containerID: containerd://f3b2e9bc393f2ecbe32bf70305c71f7c4439643d3e99d617af2e185e431a06f2
    image: docker.io/library/postgres:14
    imageID: docker.io/library/postgres@sha256:1e6c52c366e39e869184256c45757e1c85ba15b3d244b0a2cea640da6df1c4e3
    lastState: {}
    name: create-user
    ready: true
    restartCount: 1
    started: false
    state:
      terminated:
        containerID: containerd://f3b2e9bc393f2ecbe32bf70305c71f7c4439643d3e99d617af2e185e431a06f2
        exitCode: 0
        finishedAt: "2025-05-02T17:09:28Z"
        reason: Completed
        startedAt: "2025-05-02T17:09:28Z"
  phase: Running
  podIP: 10.105.1.90
  podIPs:
  - ip: 10.105.1.90
  qosClass: Burstable
  startTime: "2025-05-02T17:09:05Z"

@chuckbutkus
Copy link
Author

Here is the timing output against that specific instance (3 pods were running):

Execution time: 1.304527929998585 seconds
Execution time: 0.09320262000255752 seconds
Execution time: 0.680994501002715 seconds
Execution time: 0.06786099000601098 seconds
Execution time: 0.09207313400111161 seconds
Execution time: 0.06419829699734692 seconds
Execution time: 0.07874106900999323 seconds
Execution time: 0.059015099992393516 seconds
Execution time: 0.07837342499988154 seconds
Execution time: 0.0593133889924502 seconds
Execution time: 0.06244936700386461 seconds
Execution time: 0.07826667200424708 seconds
Execution time: 2.1241948700044304 seconds
Execution time: 0.059538718007388525 seconds
Execution time: 0.062477888001012616 seconds
Execution time: 0.06455654199817218 seconds
Execution time: 0.08736506100103725 seconds
Execution time: 0.20436898100888357 seconds
Execution time: 0.05937864699808415 seconds
Execution time: 0.07659130801039282 seconds
Execution time: 0.08491291599057149 seconds
Execution time: 0.3095227729936596 seconds
Execution time: 0.0686728540022159 seconds
Execution time: 0.22659412599750794 seconds
Execution time: 0.21168588398722932 seconds
Execution time: 0.18607174399949145 seconds
Execution time: 0.06820712800254114 seconds
Execution time: 0.07376923000265379 seconds
Execution time: 0.07464257199899293 seconds
Execution time: 1.9580987079971237 seconds
Execution time: 0.07136483599606436 seconds
Execution time: 0.057972127993707545 seconds
Execution time: 5.4687251830036985 seconds
Execution time: 0.10630319399933796 seconds
Execution time: 0.08695695700589567 seconds
Execution time: 0.18041764000372496 seconds
Execution time: 2.944205603998853 seconds
Execution time: 0.09191595599986613 seconds
Execution time: 0.07889290299499407 seconds
Execution time: 9.750665097002639 seconds
Execution time: 0.05911111399473157 seconds
Execution time: 0.19535374898987357 seconds
Execution time: 0.15692627399403136 seconds
Execution time: 0.06566001300234348 seconds
Execution time: 4.882805098008248 seconds
Execution time: 1.7401683710049838 seconds
Execution time: 1.4850838179991115 seconds
Execution time: 0.05971427600889001 seconds
Execution time: 0.05720227598794736 seconds
Execution time: 0.07368188700638711 seconds
Execution time: 0.05690579600923229 seconds
Execution time: 0.06976299099915195 seconds
Execution time: 0.05661270899872761 seconds
Execution time: 0.06507941099698655 seconds
Execution time: 5.1868801309901755 seconds
Execution time: 1.3714926499960711 seconds
Execution time: 0.0630622559983749 seconds
Execution time: 0.06139660099870525 seconds
Execution time: 3.028980664006667 seconds
Execution time: 0.05762249999679625 seconds
Execution time: 0.24831268000707496 seconds
Execution time: 0.06690425999113359 seconds
Execution time: 0.0664334909961326 seconds
Execution time: 3.5465294030000223 seconds
Execution time: 0.0634266929992009 seconds
Execution time: 0.06974554200132843 seconds
Execution time: 4.571252239999012 seconds
Execution time: 0.05839185199874919 seconds
Execution time: 4.41441675099486 seconds
Execution time: 0.05509779498970602 seconds
Execution time: 6.0068181280075805 seconds
Execution time: 0.13702252598886844 seconds
Execution time: 0.07375073501316365 seconds
Execution time: 0.08377896800811868 seconds
Execution time: 0.057485080003971234 seconds
Execution time: 2.7532506690040464 seconds

@shawkins
Copy link
Contributor
shawkins commented May 2, 2025

thank you @chuckbutkus we'll try to reproduce this. If possible can you try @ahus1 suggestion from #39023 to enabled tracing?

@ahus1 ahus1 self-assigned this May 5, 2025
@VonNao
Copy link
VonNao commented May 6, 2025

@shawkins Sorry for the late response. I updated #39023 with some screenshots from jaeger.

@ahus1
Copy link
Contributor
ahus1 commented May 7, 2025

Please follow #39023 for a resolution. Closing this one as a duplicate.

@ahus1 ahus1 closed this as completed May 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ahus1 ahus1

Labels
area/core kind/bug Categorizes a PR related to a bug team/cloud-native team/sre
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mposolda @shawkins @ahus1 @VonNao @chuckbutkus
0