This repository demonstrates how to properly implement trace context propagation when using Dapr with Apache Pulsar in a Java application. It focuses on maintaining trace context across service boundaries in a pub/sub pattern.
- Java 17 or higher
- Maven
- Docker and Docker Compose
- Dapr CLI installed and initialized
git clone https://github.com/rochabr/dapr-trace-example.git
cd dapr-trace-exampleStart Pulsar using Docker Compose:
docker-compose up -dcd subscriber
mvn clean package
dapr run --app-id subscriber \
--app-port 8081 \
--resources-path ../components \
--config ../config.yaml \
-- java -jar target/subscriber-0.0.1-SNAPSHOT.jarIn a new terminal:
cd publisher
mvn clean package
dapr run --app-id publisher \
--app-port 8082 \
--resources-path ../components \
--config ../config.yaml \
-- java -jar target/publisher-0.0.1-SNAPSHOT.jarMake a request to the publisher:
curl http://localhost:8082/publishOpen Zipkin in your browser:
http://localhost:9411
Click on "Find Traces" to see the distributed trace spanning both services.
The key to making custom trace propagation work is:
- Using OpenTelemetry in the publisher to create spans with custom trace context
- Explicitly adding the trace context to CloudEvent metadata
- Configuring Dapr properly to handle this trace context
The publisher service uses OpenTelemetry to create and manage custom trace IDs:
@GetMapping("/publish")
public Mono publishEvent() {
// Create a custom trace ID and span ID
String customTraceId = generateCustomTraceId();
String customSpanId = generateCustomSpanId();
// Create a custom span context with your trace ID
SpanContext customSpanContext = SpanContext.create(
customTraceId,
customSpanId,
TraceFlags.getSampled(),
TraceState.getDefault()
);
// Create a span with the custom context
Span span = tracer.spanBuilder("publish-event")
.setParent(Context.current().with(Span.wrap(customSpanContext)))
.setSpanKind(SpanKind.PRODUCER)
.startSpan();
try (Scope scope = span.makeCurrent()) {
// Create order data
Map order = new HashMap<>();
// ...
// CRITICAL: Add trace context to CloudEvent metadata
Map metadata = new HashMap<>();
String traceparent = String.format("00-%s-%s-%02x",
spanContext.getTraceId(),
spanContext.getSpanId(),
spanContext.getTraceFlags().asByte());
metadata.put("cloudevent.traceparent", traceparent);
// Publish with metadata
return daprClient.publishEvent(
PUBSUB_NAME,
TOPIC_NAME,
order,
metadata
).then(/* ... */);
} finally {
span.end();
}
}The publisher's pom.xml includes the necessary OpenTelemetry dependencies:
<!-- OpenTelemetry -->
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-api</artifactId>
<version>${opentelemetry.version}</version>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk</artifactId>
<version>${opentelemetry.version}</version>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-zipkin</artifactId>
<version>${opentelemetry.version}</version>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-trace</artifactId>
<version>${opentelemetry.version}</version>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-context</artifactId>
<
905E
version>${opentelemetry.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/io.opentelemetry.semconv/opentelemetry-semconv -->
<dependency>
<groupId>io.opentelemetry.semconv</groupId>
<artifactId>opentelemetry-semconv</artifactId>
<version>1.30.0</version>
</dependency>The publisher initializes OpenTelemetry in the main application class:
@SpringBootApplication
public class PublisherApplication {
public static void main(String[] args) {
// Initialize OpenTelemetry
initializeOpenTelemetry();
SpringApplication.run(PublisherApplication.class, args);
}
private static void initializeOpenTelemetry() {
// Configure OpenTelemetry
Resource resource = Resource.getDefault()
.merge(Resource.create(io.opentelemetry.api.common.Attributes.of(
ResourceAttributes.SERVICE_NAME, "publisher-service"
)));
SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
.addSpanProcessor(BatchSpanProcessor.builder(
ZipkinSpanExporter.builder()
.setEndpoint("http://localhost:9411/api/v2/spans")
.build())
.build())
.setResource(resource)
.build();
OpenTelemetrySdk sdk = OpenTelemetrySdk.builder()
.setTracerProvider(tracerProvider)
.setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
.buildAndRegisterGlobal();
}
@Bean
public Tracer tracer() {
return GlobalOpenTelemetry.getTracer("publisher-tracer");
}
}Helper methods to generate valid trace IDs and span IDs:
private String generateCustomTraceId() {
// Must be 32 hex characters (16 bytes)
String uuid = UUID.randomUUID().toString().replace("-", "");
String uuid2 = UUID.randomUUID().toString().replace("-", "");
return uuid + uuid2.substring(0, 32 - uuid.length());
}
private String generateCustomSpanId() {
// Must be 16 hex characters (8 bytes)
return UUID.randomUUID().toString().replace("-", "").substring(0, 16);
}No special Dapr configuration is needed for the trace propagation, as we're explicitly adding the trace context to the CloudEvent metadata.
The subscriber doesn't need OpenTelemetry integration since:
- Dapr properly passes the trace context through the CloudEvent
- The trace context is available in HTTP headers
- Dapr's runtime creates appropriate child spans automatically
Follow the setup instructions in the previous sections to run the example and verify trace propagation in Zipkin.
- W3C Trace Context Format: The trace context must follow the W3C format:
00-<traceId>-<spanId>-<flags> - CloudEvent Metadata: Using
cloudevent.traceparentin metadata is crucial for proper propagation - Custom Trace IDs: Creating custom span contexts allows control over trace IDs
If you see multiple traces instead of a single connected trace:
- Verify the trace context is correctly formatted in W3C format
- Check that the CloudEvent metadata contains the
cloudevent.traceparentfield - Ensure Zipkin is properly configured to receive spans