Contents
Overview
Distributed tracing is a critical feature of microservice based applications, since it traces workflow both within a service and across multiple services. This provides insight to sequence and timing data for specific blocks of work, which helps you identify performance and operational issues. Helidon includes support for distributed tracing through its own API, backed by either through the OpenTelemetry API, or by OpenTracing API. Tracing is integrated with WebServer and Security.
Maven Coordinates
To enable Helidon Tracing add the following dependency to your project’s pom.xml (see Managing Dependencies).
<dependencies>
<dependency>
<groupId>io.helidon.tracing</groupId>
<artifactId>helidon-tracing</artifactId>
</dependency>
<dependency>
<groupId>io.helidon.webserver.observe</groupId>
<artifactId>helidon-webserver-observe-tracing</artifactId>
</dependency>
</dependencies>- Helidon tracing dependency.
- Observability dependencies for tracing.
For further processing of the tracing data, different providers are used.
For Jaeger:
<dependency>
<groupId>io.helidon.tracing.providers</groupId>
<artifactId>helidon-tracing-providers-jaeger</artifactId>
</dependency>For Zipkin:
<dependency>
<groupId>io.helidon.tracing.providers</groupId>
<artifactId>helidon-tracing-providers-zipkin</artifactId>
</dependency>For OpenTelemetry:
<dependency>
<groupId>io.helidon.tracing.providers</groupId>
<artifactId>helidon-tracing-providers-opentelemetry</artifactId>
</dependency>For OpenTracing:
<dependency>
<groupId>io.helidon.tracing.providers</groupId>
<artifactId>helidon-tracing-providers-opentracing</artifactId>
</dependency>Usage
This section explains a few concepts that you need to understand before you get started with tracing.
In the context of this document, a service is synonymous with an application.
A span is the basic unit of work done within a single service, on a single host. Every span has a name, starting timestamp, and duration. For example, the work done by a REST endpoint is a span. A span is associated to a single service, but its descendants can belong to different services and hosts.
A trace contains a collection of spans from one or more services, running on one or more hosts. For example, if you trace a service endpoint that calls another service, then the trace would contain spans from both services. Within a trace, spans are organized as a directed acyclic graph (DAG) and can belong to multiple services, running on multiple hosts.
Baggage is a collection of key-value pairs associated with a span.
Span context captures data about a span not related to its duration, such as the tracer ID, the span ID, and baggage.
Support for specific tracers is abstracted. Your application can depend on the Helidon abstraction layer and provide a specific tracer implementation as a Java ServiceLoader service. Helidon provides such an implementation for:
OpenTracing tracers, either using the
GlobalTracer, provider resolver approach, or explicitly using Zipkin tracerOpenTelemetry tracers, either using the global OpenTelemetry instance, or explicitly using Jaeger tracer
Setup WebServer
TracerTracer tracer = TracerBuilder.create("helidon")
.build();
WebServer.builder()
.addFeature(ObserveFeature.builder()
.addObserver(TracingObserver.create(tracer))
.build())
.build()
.start();- Create a
Tracer. - Add an observability feature using the created
Tracer.
Creating Custom Spans
To create a custom span from tracer:
Span span = tracer.spanBuilder("name")
.tag("key", "value")
.start();
try {
// do some work
span.end();
} catch (Throwable t) {
span.end(t);
}- Create span from tracer.
- Do some work and end span.
- End span with exception.
Handling Baggage
Your application can set and read baggage associated with a Span. The Span.baggage() method returns a WritableBaggage instance.
Further, Helidon also provides read-only access to baggage linked to a SpanContext. For example, HTTP headers can convey trace ID, span ID, and baggage information and Helidon puts such information into a SpanContext. Your code can create a SpanContext from other sources as well. The SpanContext.baggage() method returns a read-only Baggage instance.
The JavaDoc for the types describes how to get and set baggage entries, get all the baggage keys, and check whether a baggage key exists in the baggage.
Responding to Span Lifecycle Events
Applications and libraries can register listeners to be notified at several moments during the lifecycle of every span:
Before a new span starts
After a new span has started
After a span ends
After a span is activated (creating a new scope)
After a scope is closed
The next sections explain how you can write and add a listener and what it can do. See the SpanListener Javadoc for more information.
Understanding What Listeners Do
A listener cannot affect the lifecycle of a span or scope it is notified about, but it can add tags and events and update the baggage associated with a span. Often a listener does additional work that does not change the span or scope such as logging a message.
When Helidon invokes the listener’s methods it passes proxies for the Span.Builder, Span, and Scope arguments. These proxies limit the access the listener has to the span builder, span, or scope, as summarized in the following table. If a listener method tries to invoke a forbidden operation, the proxy throws a SpanListener.ForbiddenOperationException and Helidon then logs a WARNING message describing the invalid operation invocation.
| Tracing type | Changes allowed |
|---|---|
Span.Builder | Add tags |
Span | Retrieve and update baggage, add events, add tags |
Scope | none |
The following tables list specifically what operations the proxies permit.
io.helidon.tracing.Span.Builder Operations| Method | Purpose | OK? |
|---|---|---|
build() | Starts the span. | - |
end methods | Ends the span. | - |
get() | Starts the span. | - |
kind(Kind) | Sets the "kind" of span (server, client, internal, etc.) | - |
parent(SpanContext) | Sets the parent of the span to be created from the builder. | - |
start() | Starts the span. | - |
start(Instant) | Starts the span. | - |
tag methods | Add a tag to the builder before the span is built. | ✓ |
unwrap(Class) | Cast the builder to the specified implementation type. † | ✓ |
† Helidon returns the unwrapped object, not a proxy for it.
io.helidon.tracing.Span Operations| Method | Purpose | OK? |
|---|---|---|
activate() | Makes the span "current", returning a Scope. | - |
addEvent methods | Associate a string (and optionally other info) with a span. | ✓ |
baggage() | Returns the Baggage instance associated with the span. | ✓ |
context() | Returns the SpanContext associated with the span. | ✓ |
status(Status) | Sets the status of the span. | - |
any tag method | Add a tag to the span. | ✓ |
unwrap(Class) | Cast the span to the specified implementation type. † | ✓ |
† Helidon returns the unwrapped object, not a proxy to it.
io.helidon.tracing.Scope Operations| Method | Purpose | OK? |
|---|---|---|
close() | Close the scope. | - |
isClosed() | Reports whether the scope is closed. | ✓ |
io.helidon.tracing.SpanContext Operations| Method | Purpose | OK? |
|---|---|---|
asParent(Span.Builder) | Sets this context as the parent of a new span builder. | ✓ |
baggage() | Returns Baggage instance associated with the span context. | ✓ |
spanId() | Returns the span ID. | ✓ |
traceId() | Returns the trace ID. | ✓ |
Adding a Listener
Explicitly Registering a Listener on a Tracer
Create a SpanListener instance and invoke the Tracer#register(SpanListener) method to make the listener known to that tracer.
Automatically Registering a Listener on all Tracer Instances
Helidon also uses Java service loading to locate listeners and register them automatically on all Tracer objects. Follow these steps to add a listener service provider.
- Implement the
SpanListenerinterface. - Declare your implementation as a service provider:
- Create the file
META-INF/services/io.helidon.tracing.SpanListenercontaining a line with the fully-qualified name of your class which implementsSpanListener. - If your service has a
module-info.javafile add the following line to it:provides io.helidon.tracing.SpanListener with <your-implementation-class>;content_copy
- Create the file
The SpanListener interface declares default no-op implementations for all the methods, so your listener can implement only the methods it needs to.
Helidon invokes each listener’s methods in the following order:
| Method | When invoked |
|---|---|
starting(Span.Builder<?> spanBuilder) | Just before a span is started from its builder. |
started(Span span) | Just after a span has started. |
activated(Span span, Scope scope) | After a span has been activated, creating a new scope. A given span might never be activated; it depends on the code. |
closed(Span span, Scope scope) | After a scope has been closed. |
ended(Span span) | After a span has ended successfully. |
ended(Span span, Throwable t) | After a span has ended unsuccessfully. |
Helidon Spans
Traced spans
The following table lists all spans traced by Helidon components:
| component | span name | description |
|---|---|---|
web-server | HTTP Request | The overall span of the Web Server from request initiation until response Note that in Zipkin the name is replaced with jax-rs span name if jax-rs tracing is used. |
web-server | content-read | Span for reading the request entity |
web-server | content-write | Span for writing the response entity |
security | security | Processing of request security |
security | security:atn | Span for request authentication |
security | security:atz | Span for request authorization |
security | security:response | Processing of response security |
security | security:outbound | Processing of outbound security |
jax-rs | A generated name | Span for the resource method invocation, name is generated from class and method name |
jax-rs | jersey-client-call | Span for outbound client call |
Some of these spans log to the span. These log events can be (in most cases) configured:
| span name | log name | configurable | enabled by default | description |
|---|---|---|---|---|
HTTP Request | handler.class | YES | YES | Each handler has its class and event logged |
security | status | YES | YES | Logs either "status: PROCEED" or "status: DENY" |
security:atn | security.user | YES | NO | The username of the user if logged in |
security:atn | security.service | YES | NO | The name of the service if logged in |
security:atn | status | YES | YES | Logs the status of security response (such as SUCCESS) |
security:atz | status | YES | YES | Logs the status of security response (such as SUCCESS) |
security:outbound | status | YES | YES | Logs the status of security response (such as SUCCESS) |
There are also tags that are set by Helidon components. These are not configurable.
| span name | tag name | description |
|---|---|---|
HTTP Request | component | name of the component - helidon-webserver, or jaxrs when using MP |
HTTP Request | http.method | HTTP method of the request, such as GET, POST |
HTTP Request | http.status_code | HTTP status code of the response |
HTTP Request | http.url | The path of the request (for SE without protocol, host and port) |
HTTP Request | error | If the request ends in error, this tag is set to true, usually accompanied by logs with details |
security | security.id | ID of the security context created for this request (if security is used) |
jersey-client-call | http.method | HTTP method of the client request |
jersey-client-call | http.status_code | HTTP status code of client response |
jersey-client-call | http.url | Full URL of the request (such as http://localhost:8080/greet) |
Configuration
The following configuration should be supported by all tracer implementations (if feasible)
Jaeger tracer configuration.
Type: io.helidon.tracing.Tracer
This is a standalone configuration type, prefix from configuration root: tracing
Configuration options
| key | type | default value | description |
|---|---|---|---|
client-cert-pem | Certificate of client in PEM format. | ||
exporter-timeout | Duration | PT10S | Timeout of exporter requests. |
max-export-batch-size | int | 512 | Maximum Export Batch Size of exporter requests. |
max-queue-size | int | 2048 | Maximum Queue Size of exporter requests. |
private-key-pem | Private key in PEM format. | ||
propagation | PropagationFormat[] (B3, B3_SINGLE, JAEGER, W3C) | JAEGER | Add propagation format to use. |
sampler-param | Number | 1 | The sampler parameter (number). |
sampler-type | SamplerType (CONSTANT, RATIO) | CONSTANT | Sampler type. See <a href="https://www.jaegertracing.io/docs/latest/sampling/#client-sampling-configuration">Sampler types</a>. |
schedule-delay | Duration | PT5S | Schedule Delay of exporter requests. |
span-processor-type | SpanProcessorType (SIMPLE, BATCH) | batch | Span Processor type used. |
trusted-cert-pem | Trusted certificates in PEM format. |
Traced Spans Configuration
Each component and its spans can be configured using Config. The traced configuration has the following layers:
TracingConfig- the overall configuration of traced components of HelidonComponentTracingConfig- a component of Helidon that traces spans (such asweb-server,security,jax-rs)SpanTracingConfig- a single traced span within a component (such assecurity:atn)SpanLogTracingConfig- a single log event on a span (such assecurity.userin spansecurity:atn)
The components using tracing configuration use the TracingConfigUtil. This uses the io.helidon.common.Context to retrieve current configuration.
Configuration Using Builder
Builder approach, example that disables a single span log event:
TracingConfig.builder()
.addComponent(ComponentTracingConfig.builder("web-server")
.addSpan(SpanTracingConfig.builder("HTTP Request")
.addSpanLog(SpanLogTracingConfig.builder("content-write")
.enabled(false)
.build())
.build())
.build())
.build();Configuration using Helidon Config
Tracing configuration can be defined in a config file.
tracing:
components:
web-server:
spans:
- name: "HTTP Request"
logs:
- name: "content-write"
enabled: falseTracer tracer = TracerBuilder.create(config.get("tracing")).build();
server.addFeature(ObserveFeature.builder()
.addObserver(TracingObserver.create(tracer))
.build());- Create
TracerusingTracerBuilderfrom configuration. - Add the
Traceras an observability feature.
Path-based Configuration in Helidon WebServer
For Web Server we have path-based support for configuring tracing, in addition to the configuration described above.
Configuration of path can use any path string supported by the WebServer. The configuration itself has the same possibilities as traced configuration described above. The path-specific configuration will be merged with global configuration (path is the "newer" configuration, global is the "older")
tracing:
paths:
- path: "/favicon.ico"
enabled: false
- path: "/metrics"
enabled: false
- path: "/health"
enabled: false
- path: "/greet"
components:
web-server:
spans:
- name: "content-read"
new-name: "read"
enabled: falseRenaming top level span using request properties
To have a nicer overview in search pane of a tracer, you can customize the top-level span name using configuration.
Example:
tracing:
components:
web-server:
spans:
- name: "HTTP Request"
new-name: "HTTP %1$s %2$s"This is supported ONLY for the span named "HTTP Request" on component "web-server".
Parameters provided:
- Method - HTTP method
- Path - path of the request (such as '/greet')
- Query - query of the request (may be null)
Additional Information
WebClient Span Propagation
Span propagation is supported with Helidon WebClient. Tracing propagation is automatic as long as the current span context is available in Helidon Context (which is automatic when running within a WebServer request).
<dependencies>
<dependency>
<groupId>io.helidon.webclient</groupId>
<artifactId>helidon-webclient</artifactId>
</dependency>
<dependency>
<groupId>io.helidon.webclient</groupId>
<artifactId>helidon-webclient-tracing</artifactId>
</dependency>
</dependencies>WebClient client = WebClient.builder()
.addService(WebClientTracing.create())
.build();
String response = client.get()
.uri(uri)
.requestEntity(String.class);Jaeger Tracing
<dependency>
<groupId>io.helidon.tracing</groupId>
<artifactId>helidon-tracing-providers-jaeger</artifactId>
</dependency>Configuring Jaeger
Jaeger tracer configuration.
Type: io.helidon.tracing.Tracer
This is a standalone configuration type, prefix from configuration root: tracing
Configuration options
| key | type | default value | description |
|---|---|---|---|
client-cert-pem | Certificate of client in PEM format. | ||
exporter-timeout | Duration | PT10S | Timeout of exporter requests. |
max-export-batch-size | int | 512 | Maximum Export Batch Size of exporter requests. |
max-queue-size | int | 2048 | Maximum Queue Size of exporter requests. |
private-key-pem | Private key in PEM format. | ||
propagation | PropagationFormat[] (B3, B3_SINGLE, JAEGER, W3C) | JAEGER | Add propagation format to use. |
sampler-param | Number | 1 | The sampler parameter (number). |
sampler-type | SamplerType (CONSTANT, RATIO) | CONSTANT | Sampler type. See <a href="https://www.jaegertracing.io/docs/latest/sampling/#client-sampling-configuration">Sampler types</a>. |
schedule-delay | Duration | PT5S | Schedule Delay of exporter requests. |
span-processor-type | SpanProcessorType (SIMPLE, BATCH) | batch | Span Processor type used. |
trusted-cert-pem | Trusted certificates in PEM format. |
The following is an example of a Jaeger configuration, specified in the YAML format.
tracing:
service: "helidon-full-http"
protocol: "https"
host: "jaeger"
port: 14240Jaeger Tracing Metrics
As the Jaeger Tracing section describes, you can use Jaeger tracing in your Helidon application.
Zipkin Tracing
<dependency>
<groupId>io.helidon.tracing.providers</groupId>
<artifactId>helidon-tracing-providers-zipkin</artifactId>
</dependency>Configuring Zipkin
Zipkin tracer configuration
Type: io.opentracing.Tracer
This is a standalone configuration type, prefix from configuration root: tracing
Configuration options
| key | type | default value | description |
|---|---|---|---|
api-version | Version (V1, V2) | V2 | Version of Zipkin API to use. Defaults to Version#V2. |
The following is an example of a Zipkin configuration, specified in the YAML format.
tracing:
zipkin:
service: "helidon-service"
protocol: "https"
host: "zipkin"
port: 9987
api-version: 1
# this is the default path for API version 2
path: "/api/v2/spans"
tags:
tag1: "tag1-value"
tag2: "tag2-value"
boolean-tags:
tag3: true
tag4: false
int-tags:
tag5: 145
tag6: 741Example of Zipkin trace:
