Tags: libaineu2004/brod
Tags
Introduce roundrobin_v2 group protocol (brod version 3.3.3)
Prior to this change, brod's 'roundrobin' group protocol commits
acknowledged offsets to kafka, then +1 as begin-offset when assign
to group members to continue after restart/rebalance.
Starting from this version, brod will commit 'next offset to fetch',
to be compliant with the protocol spec and other kafka client
implementations, as well as monitoring tools.
For offsets committed to kafka:
'roundrobin_v2' will upgrade old commits to new version by +1.
This is done by making use of the metadata committed together
with the offsets:
'roundrobin' metadata should be a string containing the Erlang node
name and the group coordinator pid. e.g. "myconsumer@myhost/<0,6,0>"
'roundrogin_v2' uses the same but with a "+1/" prefix.
NOTE: Since it's a new protocol, it does not support rolling upgrade
from a running 'roundrobin' group. i.e. all 'roundrobin' members
will have to be stopped before starting 'roundrobin_v2' members.
Nonetheless, there is no risk even if 'roudrobin' and
'roundrobin_v2' members are started at the same time,
because kafka would not allow a member supporting different
protocol to join an active group.
It is however NOT OK for 'roundrobin' members to pick up
'roundrobin_v2' commits. i.e. downgrade is not supported.
For consumer managed offsets (offset_commit_policy = consumer_managed):
There is no easy way to smooth upgrade consumer managed offsets,
therefor we will keep commiting 'acked offsets' instead of
'begin offset' until next major release.
Add option to allow topic subscriber handling Kafka message or messag… …e_set (kafka4beam#229) * Add option to allow topic subscriber handling Kafka message or message_set * Add option to allow group subscriber handling Kafka message or message_set * Bump version to 3.1.0
Merge pull request kafka4beam#219 from klarna/use-kafka-protocol-1.0 Make it work with kafka_protocol 1.0.0
Support sasl-plain username password in text file (kafka4beam#216)
PreviousNext