This is a Kafka client written in pure Rust.
It uses kafka-protocol for the protocol implementation, and tokio for async io.
- Multiplexed, async IO
- Client-side load balancing
- Connection retry with exponential backoff
- Generic over the IO channel
- Custom partitioning strategies
-
Producer
- Implement automatic retry in a way that can guarantee message order
- Implement idempotent producer
- Implement transactions
-
Consumer
- Figure out how groups actually work
-
Respect the throttle time returned by the server.
-
Other questions:
- What is the difference between
offsetandsequence 52C3in the context of aProduceRequest?
- What is the difference between
-
Benchmarking
- See if producer can hit 800k records/s: https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
- Update: we are pretty close I think. I did a local benchmark with random data, and hit 1M records/s. See the
ProduceRandomimplementation incmd/producer.rsfor how that works. - It seems like sending to one partition is faster than multiple right now.
-
More tracing
-
More tests