Closed
Description
Is there an existing issue for this?
- I have searched the existing issues
What happened?
Cilium logs are flooded with "hubble events queue is full" warnings. These should be rate-limited and/or aggregated for longer time period than the until the consumer recovers. Should also consider if this should be at warning level, or if info or debug would be sufficient.
One downside of this that these messages make it harder to see more relevant warnings in the logs, e.g.:
Top 5 errors/warnings:
hubble events queue is full; dropping messages
hubble events queue is processing messages again: 1 messages were lost
hubble events queue is processing messages again: 2 messages were lost
hubble events queue is processing messages again: 3 messages were lost
hubble events queue is processing messages again: 4 messages were lost
Cilium Version
master
Kernel Version
N/A
Kubernetes Version
N/A
Sysdump
No response
Relevant log output
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 12 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 1 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 3 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 7 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 12 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 4 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 1 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 202 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 4 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 1 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 8 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 3 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 1 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 1 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 1 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 1 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 7 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 7 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 1 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 1 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 1 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 1 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 8 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 6 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 1 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 1 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 39 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 8 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 6 messages were lost" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is full; dropping messages" subsys=hubble
Mar 20 19:04:19 runtime cilium-agent[33599]: level=warning msg="hubble events queue is processing messages again: 4 messages were lost" subsys=hubble
...Anything else?
No response
Code of Conduct
- I agree to follow this project's Code of Conduct