Kafka consumer: per-partition size-limited queue for backpressure, configurable fetch_max_wait #997
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
This is a refactoring of Kafka consumer which aims to simplify the code and avoid SetOffsets call during poll loop.
We also tune fetch max wait time here to be 500ms instead of 5000ms since there is a chance that loaded partition may be paused until fetch returns (if no other partitions are active at this point - fetch blocks for up to 5 seconds).
With the changes made PR improves consumer throughput (according to added benchmark) in 3 times, having 30k msgs per seconds instead of 10k msgs per second (mostly due to fetch max wait time tuning).
New options:
fetch_max_wait- duration, default 500mspartition_queue_max_size- int, default 1000