spdk_tgt encountered an assertion failure in longhorn-spdk-helper during a CI test run #10599Description
Describe the Bug
spdk_tgt encountered an assertion failure in longhorn-spdk-helper during a CI test run.
...
[2025-03-18 02:29:52.317504] thread.c:2396:put_io_channel: *DEBUG*: Releasing io_channel 0x1dac1800 for io_device bdev_test-vol-0-replica-3-snap-rebuild-3c7680fbn1 (0x202c5811) on thread app_thread
[2025-03-18 02:29:52.317507] bdev.c:4653:bdev_channel_destroy: *DEBUG*: Destroying channel 0x1dac1860 for bdev test-vol-0-replica-3-snap-rebuild-3c7680fbn1 on thread 0x1d965ac0
[2025-03-18 02:29:52.317513] thread.c:2460:spdk_put_io_channel: *DEBUG*: Putting io_channel 0x20279870 for io_device test-vol-0-replica-3-snap-rebuild-3c7680fbn1 (0x202c5810) on thread app_thread refcnt 1
[2025-03-18 02:29:52.317516] thread.c:2460:spdk_put_io_channel: *DEBUG*: Putting io_channel 0x1dab82b0 for io_device accel (0xd751a0) on thread app_thread refcnt 21
spdk_tgt: bdev.c:3966: bdev_channel_destroy_resource: Assertion `TAILQ_EMPTY(&ch->io_submitted)' failed.
To Reproduce
Execute the unit test case TestSPDKMultipleThreadSnapshotOpsAndRebuilding
Expected Behavior
Should pass without error
Support Bundle for Troubleshooting
https://drive.google.com/file/d/1HAgqIpxifKZIYtUU6zXMLID3R0Ua6d3S/view?usp=sharing
Environment
- Longhorn version:
- Impacted volume (PV):
- Installation method (e.g. Rancher Catalog App/Helm/Kubectl):
- Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version:
- Number of control plane nodes in the cluster:
- Number of worker nodes in the cluster:
- Node config
- OS type and version:
- Kernel version:
- CPU per node:
- Memory per node:
- Disk type (e.g. SSD/NVMe/HDD):
- Network bandwidth between the nodes (Gbps):
- Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
- Number of Longhorn volumes in the cluster:
Additional context
No response
Workaround and Mitigation
No response