Rotel is a lightweight OpenTelemetry collector implemented in Rust.
Rotel provides an efficient, high-performance solution for collecting, processing, and exporting telemetry data. Rotel is ideal for resource-constrained environments and applications where minimizing overhead is critical.
Features:
- Supports metrics, logs, and traces
- OTLP receiver supporting gRPC, HTTP/Protobuf, and HTTP/JSON
- OTLP exporter supporting gRPC and HTTP/Protobuf
- Built-in batching and retry mechanisms
- Experimental Clickhouse, Datadog, and Kafka exporters
Rotel can be easily bundled with popular runtimes as packages. Its Rust implementation ensures minimal resource usage and a compact binary size, simplifying deployment without the need for a sidecar container.
Runtime integrations:
- Python: streamfold/pyrotel
- Node.js: streamfold/rotel-nodejs
We also have a custom AWS Lambda Extension layer that embeds the Rotel collector to provide an OpenTelemetry collector with minimal coldstart latency:
- Rotel Lambda Extension: streamfold/rotel-lambda-extension
Rotel is fully open-sourced and licensed under the Apache 2.0 license.
Rotel is currently in early development, and we are actively looking for feedback from the community. It is not recommended for production use at this time.
To quickly get started with Rotel you can leverage the bundled Python or NodeJS packages. Or if you'd like to use Rotel directly from Docker, follow these steps:
-
Running Rotel
- We use the prebuilt docker image for this example, but you can also download a binary from the releases page.
- Execute Rotel with the following arguments. To debug metrics or logs, add
an additional
--debug-log metrics|logs.
docker run -ti -p 4317-4318:4317-4318 streamfold/rotel --debug-log traces --exporter blackhole
- Rotel is now listening on localhost:4317 (gRPC) and localhost:4318 (HTTP).
-
Verify
- Send OTLP traces to Rotel and verify that it is receiving data:
go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@latest telemetrygen traces --otlp-insecure --duration 5s
- Check the output from Rotel and you should see several "Received traces" log lines.
Rotel is configured on the command line with multiple flags. See the table below for the full list of options. Rotel will also output the full argument list:
rotel start --helpAll CLI arguments can also be passed as environment variable by prefixing with ROTEL_ and switching hyphens to
underscores. For example, --otlp-grpc-endpoint localhost:5317 can also be specified by setting the environment
variable ROTEL_OTLP_GRPC_ENDPOINT=localhost:5317.
Any option above that does not contain a default is considered false or unset by default.
| Option | Default | Options |
|---|---|---|
| --daemon | ||
| --log-format | text | json |
| --pid-file | /tmp/rotel-agent.pid | |
| --log-file | /tmp/rotel-agent.log | |
| --debug-log | metrics, traces, logs | |
| --debug-log-verbosity | basic | basic, detailed |
| --otlp-grpc-endpoint | localhost:4317 | |
| --otlp-http-endpoint | localhost:4318 | |
| --otlp-grpc-max-recv-msg-size-mib | 4 | |
| --exporter | otlp | otlp, blackhole, datadog, clickhouse, aws-xray, kafka |
| --otlp-receiver-traces-disabled | ||
| --otlp-receiver-metrics-disabled | ||
| --otlp-receiver-logs-disabled | ||
| --otlp-receiver-traces-http-path | /v1/traces | |
| --otlp-receiver-metrics-http-path | /v1/metrics | |
| --otlp-receiver-logs-http-path | /v1/logs | |
| --otel-resource-attributes | ||
| --enable-internal-telemetry |
The PID and LOG files are only used when run in --daemon mode.
See the section for Multiple Exporters for how to configure multiple exporters
The OTLP exporter is the default, or can be explicitly selected with --exporter otlp.
| Option | Default | Options |
|---|---|---|
| --otlp-exporter-endpoint | ||
| --otlp-exporter-protocol | grpc | grpc, http |
| --otlp-exporter-custom-headers | ||
| --otlp-exporter-compression | gzip | gzip, none |
| --otlp-exporter-authenticator | sigv4auth | |
| --otlp-exporter-tls-cert-file | ||
| --otlp-exporter-tls-cert-pem | ||
| --otlp-exporter-tls-key-file | ||
| --otlp-exporter-tls-key-pem | ||
| --otlp-exporter-tls-ca-file | ||
| --otlp-exporter-tls-ca-pem | ||
| --otlp-exporter-tls-skip-verify | ||
| --otlp-exporter-request-timeout | 5s | |
| --otlp-exporter-retry-initial-backoff | 5s | |
| --otlp-exporter-retry-max-backoff | 30s | |
| --otlp-exporter-retry-max-elapsed-time | 300s |
Any of the options that start with --otlp-exporter* can be set per telemetry type: metrics, traces or logs. For
example, to set a custom endpoint to export traces to, set: --otlp-exporter-traces-endpoint. For other telemetry
types their value falls back to the top-level OTLP exporter config.
The Rotel OTLP exporter can export to the Cloudwatch OTLP endpoints for traces and logs. You'll need to select the HTTP protocol and enable the sigv4auth authenticator.
The sigv4auth authenticator requires the AWS authentication environment variables to be set.
Traces
Tracing requires that you enable Transaction Search in the AWS console before you can send OTLP traces.
Here is the full environment variable configuration to send traces to Cloudwatch, swap the region code as needed.
ROTEL_EXPORTER=otlp
ROTEL_OTLP_EXPORTER_PROTOCOL=http
ROTEL_OTLP_EXPORTER_TRACES_ENDPOINT=https://xray.<region code>.amazonaws.com/v1/traces
ROTEL_OTLP_EXPORTER_AUTHENTICATOR=sigv4authLogs
To send OTLP logs to Cloudwatch you must create a log group and log stream. Exporting will fail if these do not exist ahead of time and they are not created by default.
Here is the full environment variable configuration to send logs to Cloudwatch, swap the region code and log group/stream as needed.
ROTEL_EXPORTER=otlp
ROTEL_OTLP_EXPORTER_PROTOCOL=http
ROTEL_OTLP_EXPORTER_TRACES_ENDPOINT=https://logs.<region code>.amazonaws.com/v1/logs
ROTEL_OTLP_EXPORTER_LOGS_CUSTOM_HEADERS="x-aws-log-group=<log group>,x-aws-log-stream=<log stream>"
ROTEL_OTLP_EXPORTER_AUTHENTICATOR=sigv4authThe Datadog exporter can be selected by passing --exporter datadog. The Datadog exporter only supports traces at the
moment. For more information, see the Datadog Exporter docs.
| Option | Default | Options |
|---|---|---|
| --datadog-exporter-region | us1 | us1, us3, us5, eu, ap1 |
| --datadog-exporter-custom-endpoint | ||
| --datadog-exporter-api-key |
Specifying a custom endpoint will override the region selection.
The Clickhouse exporter can be selected by passing --exporter clickhouse. The Clickhouse exporter supports metrics,
logs,
and traces.
| Option | Default | Options |
|---|---|---|
| --clickhouse-exporter-endpoint | ||
| --clickhouse-exporter-database | otel | |
| --clickhouse-exporter-table-prefix | otel | |
| --clickhouse-exporter-compression | lz4 | none, lz4 |
| --clickhouse-exporter-async-insert | true | true, false |
| --clickhouse-exporter-enable-json | ||
| --clickhouse-exporter-json-underscore | ||
| --clickhouse-exporter-user | ||
| --clickhouse-exporter-password |
The Clickhouse endpoint must be specified while all other options can be left as defaults. The table prefix is prefixed
onto the specific telemetry table name with underscore, so a table prefix of otel will be combined with _traces to
generate the full table name of otel_traces.
The Clickhouse exporter will enable async inserts by
default,
although it can be disabled server-side. Async inserts are
recommended for most workloads to avoid overloading Clickhouse with many small inserts. Async inserts can be disabled by
specifying:
--clickhouse-exporter-async-insert false.
The exporter will not generate the table schema if it does not exist. Use the clickhouse-ddl command for generating the necessary table DDL for Clickhouse. The DDL matches the schema used in the OpenTelemetry Clickhouse exporter.
Enabling JSON via the --clickhouse-exporter-enable-json will use the new
JSON data type in Clickhouse. This data
type is only available on the most recent versions of Clickhouse. Make sure that you enable JSON with --enable-json
when creating tables with clickhouse-ddl. By default, any JSON key inserted with a period in it will create
a nested JSON object. You can replace periods in JSON keys with underscores by passing the option
--clickhouse-exporter-json-underscore which will keep the JSON keys flat. For example, the resource attribute
service.name will be inserted as service_name.
The Clickhouse exporter is built using code from the official Rust clickhouse-rs crate.
The AWS X-Ray exporter can be selected by passing --exporter aws-xray. The X-Ray exporter only supports traces. Note:
X-Ray
limits batch sizes to 50 traces segments. If you assign a --batch-max-size of greater than 50, Rotel will override and
enforce the max
batch size of 50 with the warning
INFO AWS X-Ray only supports a batch size of 50 segments, setting batch max size to 50
AWS Credentials including AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN for the X-Ray exporter
are
automatically sourced from Rotel's environment on startup.
| Option | Default | Options |
|---|---|---|
| --xray-exporter-region | us-east-1 | aws region codes |
| --xray-exporter-custom-endpoint |
For a list of available AWS X-Ray region codes here: https://docs.aws.amazon.com/general/latest/gr/xray.html
In order to run the Kafka exporter you will need to build with the --features rdkafka option. i.e. cargo build --release --features rdkafka
The Kafka exporter can be selected by passing --exporter kafka. The Kafka exporter supports metrics,
logs, and traces.
| Option | Default | Options |
|---|---|---|
| --kafka-exporter-brokers | localhost:9092 | |
| --kafka-exporter-traces-topic | otlp_traces | |
| --kafka-exporter-metrics-topic | otlp_metrics | |
| --kafka-exporter-logs-topic | otlp_logs | |
| --kafka-exporter-format | protobuf | json, protobuf |
| --kafka-exporter-compression | none | gzip, snappy, lz4, zstd, none |
| --kafka-exporter-request-timeout | 30s | |
| --kafka-exporter-acks | one | none, one, all |
| --kafka-exporter-client-id | rotel | |
| --kafka-exporter-max-message-bytes | 1000000 | |
| --kafka-exporter-linger-ms | 5 | |
| --kafka-exporter-retries | 2147483647 | |
| --kafka-exporter-retry-backoff-ms | 100 | |
| --kafka-exporter-retry-backoff-max-ms | 1000 | |
| --kafka-exporter-message-timeout-ms | 300000 | |
| --kafka-exporter-request-timeout-ms | 30000 | |
| --kafka-exporter-batch-size | 1000000 | |
| --kafka-exporter-partitioner | consistent-random | consistent, consistent-random, murmur2-random, murmur2, fnv1a, fnv1a-random |
| --kafka-exporter-partition-metrics-by-resource-attributes | false | |
| --kafka-exporter-partition-logs-by-resource-attributes | false | |
| --kafka-exporter-custom-config | ||
| --kafka-exporter-sasl-username | ||
| --kafka-exporter-sasl-password | ||
| --kafka-exporter-sasl-mechanism | PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 | |
| --kafka-exporter-security-protocol | PLAINTEXT | PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL |
The Kafka broker addresses must be specified (comma-separated for multiple brokers). The exporter will create separate topics for traces, metrics, and logs. Data can be serialized as JSON or Protobuf format.
The --kafka-exporter-acks option controls the producer acknowledgement behavior, balancing between performance and
durability:
none(acks=0): No acknowledgement required - fastest performance but least durable, data may be lost if the leader failsone(acks=1): Wait for leader acknowledgement only - balanced approach, good performance with reasonable durability (default)all(acks=all): Wait for all in-sync replicas to acknowledge - slowest but most durable, ensures data is not lost
For secure connections, you can configure SASL authentication:
rotel start --exporter kafka \
--kafka-exporter-brokers "broker1:9092,broker2:9092" \
--kafka-exporter-sasl-username "your-username" \
--kafka-exporter-sasl-password "your-password" \
--kafka-exporter-sasl-mechanism "SCRAM-SHA-256" \
--kafka-exporter-security-protocol "SASL_SSL" \
--kafka-exporter-compression "gzip" \
--kafka-exporter-acks "all"The Kafka exporter provides several options for tuning producer performance and reliability:
--kafka-exporter-linger-ms: Delay in milliseconds to wait for messages to accumulate before sending. Higher values improve batching efficiency but increase latency.--kafka-exporter-retries: How many times to retry sending a failing message. High values ensure delivery but may cause reordering.--kafka-exporter-retry-backoff-ms: Initial backoff time before retrying a failed request.--kafka-exporter-retry-backoff-max-ms: Maximum backoff time for exponentially backed-off retry requests.--kafka-exporter-message-timeout-ms: Maximum time to wait for messages to be sent successfully. Messages exceeding this timeout will be dropped.--kafka-exporter-request-timeout-ms: Timeout for individual requests to the Kafka brokers.--kafka-exporter-batch-size: Maximum size of message batches in bytes. Larger batches improve throughput but increase memory usage.--kafka-exporter-partitioner: Controls how messages are distributed across partitions. Options include consistent hashing and murmur2/fnv1a hash algorithms.
For improved consumer parallelism and data organization, you can enable custom partitioning based on telemetry data:
--kafka-exporter-partition-metrics-by-resource-attributes: When enabled, metrics are partitioned by resource attributes (like service name), grouping related metrics together.--kafka-exporter-partition-logs-by-resource-attributes: When enabled, logs are partitioned by resource attributes, organizing logs by service or application.
These options override the global partitioner setting for specific telemetry types when enabled.
For advanced use cases, you can set arbitrary Kafka producer configuration parameters using the
--kafka-exporter-custom-config option. This accepts comma-separated key=value pairs:
rotel start --exporter kafka \
--kafka-exporter-custom-config "enable.idempotence=true,max.in.flight.requests.per.connection=1" \
--kafka-exporter-brokers "broker1:9092,broker2:9092"Configuration Precedence: Custom configuration parameters are applied after all built-in options, meaning they will override any conflicting built-in settings. For example:
# The custom config will override the built-in batch size setting
rotel start --exporter kafka \
--kafka-exporter-batch-size 500000 \
--kafka-exporter-custom-config "batch.size=2000000"
# Final batch.size will be 2000000, not 500000This allows you to configure any rdkafka producer parameter that isn't explicitly exposed through dedicated CLI options. See the librdkafka configuration documentation for all available parameters.
The Kafka exporter uses the high-performance rdkafka library and includes built-in retry logic and error handling.
To run integration tests that verify actual Kafka functionality:
# Start test environment
./scripts/kafka-test-env.sh start
# Run integration tests
cargo test --test kafka_integration_tests --features integration-tests
# Stop test environment
./scripts/kafka-test-env.sh stopSee KAFKA_INTEGRATION_TESTS.md for detailed testing instructions.
You can configure the properties of the batch processor, controlling both the size limit of the batch and how long the
batch
is kept before flushing. The batch properties behave the same regardless of which exporter you use. You can override the
batch settings specifically for a telemetry type by prefixing any of the options below with the telemetry type (metrics,
logs,
or traces). For example, --traces-batch-max-size will override the batch max size for traces only.
| Option | Default | Options |
|---|---|---|
| --batch-max-size | 8192 | |
| --batch-timeout | 200ms |
Rotel also supports setting or overwriting resource attributes on OpenTelemetry logs, metrics, and traces via the
command line or environment. The --otel-resource-attributes flag accepts a comma-separated list of key-value pairs to
upsert on the
the resource
attributes of ResourceLogs, ResourceMetrics, and ResourceSpans.
For example starting rotel with the following command line argument will append or overwrite the service.name
and environment attributes. --otel-resource-attributes "service.name=my-service,environment=production"
Alternatively you can use the ROTEL_OTEL_RESOURCE_ATTRIBUTES environment variable to achieve the same outcome.
ROTEL_OTEL_RESOURCE_ATTRIBUTES=service.name=my-service,environment=production rotel start --otlp-exporter-endpoint <endpoint url>
You can override the default request timeout of 5 seconds for the OTLP Exporter with the exporter setting:
--otlp-exporter-request-timeout: Takes a string time duration, so"250ms"for 250 milliseconds,"3s"for 3 seconds, etc.
Requests will be retried if they match retryable error codes like 429 (Too Many Requests) or timeout. You can control the behavior with the following exporter options:
--otlp-exporter-retry-initial-backoff: Initial backoff duration--otlp-exporter-retry-max-backoff: Maximum backoff interval--otlp-exporter-retry-max-elapsed-time: Maximum wall time a request will be retried for until it is marked as permanent failure
All options should be represented as string time durations.
Rotel records a number of internal metrics that can help observe Rotel behavior during runtime. This telemetry is
opt-in and must be enabled with --enable-internal-telemetry. Telemetry is sent to the OTLP exporter metric endpoint
that you have configured.
NOTE: Internal telemetry is not sent to any outside sources and you are in full control of where this data is exported to.
Rotel can be configured to support exporting to multiple destinations across multiple exporter types.
The following additional configuration parameters set up support for multiple exporters. Similar to the options above,
all
CLI arguments can be passed as environment variables as well. It is not possible to set --exporter and --exporters
at the same time.
| Option | Default | Options |
|---|---|---|
| --exporters | name:type pairs, comma-separated | |
| --exporters-traces | exporter name | |
| --exporters-metrics | exporter name | |
| --exporters-logs | exporter name |
First start by defining the set of exporters that you would like to use, optionally specifying a custom name for them
to differentiate their configuration options. For example, to export logs and metrics to two separate Clickhouse nodes
while exporting traces to Datadog, we'll use the following --exporters argument (or ROTEL_EXPORTERS envvar):
--exporters logging:clickhouse,stats:clickhouse,datadogThe argument form of --exporters takes name:type pairs separated by commas, where the first part is a custom name
and
the second part is the type of exporter. You can exclude the name if there is a single exporter by that name, which
means
the name is the same as the exporter type.
Second, you then must set environment variables of the form ROTEL_EXPORTER_{NAME}_{PARAMETER} to configure the
multiple
exporters. These variable names are dynamic and use the custom name to differentiate settings for similar exporter
types.
Therefore, there are no CLI argument alternatives for them at the moment. The {PARAMETER} fields match the
configuration
options for the given exporter type.
Using our example above, the user must set, at a minimum, the following environment variables. (For Clickhouse Cloud you would need to include a username/password, but we are skipping those for brevity.)
ROTEL_EXPORTER_LOGGING_ENDPOINT=https://xxxxxxx.us-east-1.aws.clickhouse.cloud:8443ROTEL_EXPORTER_STATS_ENDPOINT=https://xxxxxxx.us-west-1.aws.clickhouse.cloud:8443ROTEL_EXPORTER_DATADOG_API_KEY=dd-abcd1234
Lastly, the user would need to connect these exporters to the telemetry types. Using the requirements above, the user would specify the following:
--exporters-traces datadog --exporters-metrics stats --exporters-logs loggingAlternatively, the following environment variables would do the same:
ROTEL_EXPORTERS_TRACES=datadogROTEL_EXPORTERS_METRICS=statsROTEL_EXPORTERS_LOGS=logging
NOTE: At the moment, only a single exporter can be set for any telemetry type. This constraint will be relaxed in the future.
The following example demonstrates how to send OTLP data to Axiom. Set your Axiom API key in the
envvar AXIOM_API_KEY and the dataset in AXIOM_DATASET:
ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS="Authorization=Bearer ${AXIOM_API_KEY},X-Axiom-Dataset=${AXIOM_DATASET}" \
./rotel start --otlp-exporter-endpoint https://api.axiom.co --otlp-exporter-protocol httpIn another window run the telemetry generator again:
telemetrygen traces --otlp-insecure --duration 1sYou should see demo trace data show up in Axiom.
Rotel includes a Python processor SDK that allows you to write custom processors in Python to modify OpenTelemetry data in-flight. The SDK provides interfaces for processing both traces and logs data (metrics coming soon!) through a simple Python API.
The processor SDK enables you to:
- Access and modify trace spans, including span data, attributes, events, links and status
- Process log records, including severity, body content, and associated attributes
- Modify resource attributes across traces and logs
- Transform data using custom Python logic before it is exported
The processor SDK also includes LSP support for code completion, syntax highlighting and marking of warnings for use in your preferred editor such as VSCode, Nvim, and Pycharm. The LSP integration is hosted on pypi and can be found at https://pypi.org/project/rotel-sdk/.
To install the sdk simply type pip install rotel-sdk
Example of a simple trace processor:
from rotel_sdk.open_telemetry.common.v1 import KeyValue, AnyValue
from rotel_sdk.open_telemetry.trace.v1 import ResourceSpans
def process_spans(resource_spans):
for scope_spans in resource_spans.scope_spans:
for span in scope_spans.spans:
# Add custom attribute to all spans
span.attributes.append(KeyValue("processed.by", AnyValue("my_processor")))Rotel also ships with prebuilt processors written in Python that you can use right out of the box or modify. Prebuilt processors are found in the processors folder under the rotel_python_processor_sdk directory.
Current prebuilt processors include...
| Name | Supported telemetry types |
|---|---|
| Attributes Processor | logs, metrics, traces, |
| Redaction Processor | logs, metrics, traces |
The SDK is built using PyO3, a robust Rust binding for Python that enables seamless interoperability between Rust and Python code. This architecture provides several benefits:
- High Performance: The core data structures remain in Rust memory while exposing a Python-friendly interface, minimizing overhead from data copying and conversions
- Memory Safety: Rust's ownership model and thread safety guarantees are preserved while allowing safe Python access to the data
- Type Safety: PyO3's type system ensures reliable conversions between Rust and Python types
- GIL Management: Automatic handling of Python's Global Interpreter Lock (GIL) for optimal performance in threaded environments
The SDK handles all the necessary conversions between Rust and Python types, making it easy to integrate Python processing logic into your Rotel collector pipeline. This allows for flexible data transformation and enrichment without modifying the core collector code.
For detailed documentation, examples, and a complete guide to writing processors, see the Python Processor SDK Documentation.
We have taken the OpenTelemetry Collector benchmark suite and adapted it to run against Rotel. You can find the testing framework at rotel-otel-loadtests and the benchmark results here. The benchmarks are run nightly comparing the latest OTEL version against the latest Rotel release.
If you set the option --debug-log to ["traces"], or the environment variable ROTEL_DEBUG_LOG=traces, then
rotel will log a summary to the log file /tmp/rotel-agent.log each time it processes trace spans. You can add also
specify metrics to debug metrics and logs to debug logs. By default the debug logging will output a single line
summary of the telemetry. You can increase the verbosity by specifying --debug-log-verbosity detailed, which will
include verbose multi-line output.
Separate from the telemetry logging, Rotel's default log level is set to INFO and can be changed with the environment
variable RUST_LOG. For example, setting RUST_LOG=debug will increase the verbosity of all logging to debug level.
This
may include logging from third-party crates used in Rotel.
On release, Rotel images are published to Dockerhub with the following tags:
streamfold/rotel:<release name>streamfold/rotel:lateststreamfold/rotel:sha-<sha>
When running an image, map the OTLP receiver ports to their local values with the flag -p 4317-4318:4317-4318.
Want to chat about this project, share feedback, or suggest improvements? Join our Discord server! Whether you're a user of this project or not, we'd love to hear your thoughts and ideas. See you there! 🚀
See the DEVELOPING.md doc for building and development instructions.