Kafka Exporter #131
Merged
Kafka Exporter #131
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mheffner
reviewed
Jul 3, 2025
mheffner
approved these changes
Jul 3, 2025
README.md
Outdated
| | --kafka-exporter-traces-topic | otlp_traces | | | ||
| | --kafka-exporter-metrics-topic | otlp_metrics | | | ||
| | --kafka-exporter-logs-topic | otlp_logs | | | ||
| | --kafka-exporter-format | json | json, protobuf | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Completes: STR-3448
Closes: #122
Adds support for the Kafka exporter. We're currently marking experimental. On the configuration side we have exposed all of the most important and high priority configuration parameters as marked https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html#autotoc_md93 and tried to maintain parity with the reference collector.
Additionally the user can set and override any librdkafka option using the
kafka_exporter_custom_configoption, exposing all potential options.There is a significant amount of test coverage and in addition to unit tests this PR features integration tests, which are placed behind a feature flag. The integration test covers advanced functionality including message key based partitioning to verify the partition by resource attributes features for metrics and logs are spread consistently and randomly across Kafka partitions.
There's a few other things we'll want to follow up with.
topic_from_metadata_keytopic_from_attribute. We should investigate how important these are before implementingpoll_readyAPI to signal backpressure to the pipeline. We need to spend further time investigating the best tokio patterns for working with rdkafka.In terms of throughput and resource utilization, values are inline with current exporters and we don't see any crazy high amounts of resources or diminished throughput when testing with loadtester at high volumes.. e.g 400K span/sec etc.