Contents

Overview

Distributed tracing is a critical feature of microservice based applications, since it traces workflow both within a service and across multiple services. This provides insight to sequence and timing data for specific blocks of work, which helps you identify performance and operational issues. Helidon includes support for distributed tracing through its own API, backed by either through the OpenTelemetry API, or by OpenTracing API. Tracing is integrated with WebServer and Security.

Maven Coordinates

To enable Helidon Tracing add the following dependency to your project’s pom.xml (see Managing Dependencies).

<dependencies>
    <dependency>
        <groupId>io.helidon.tracing</groupId>
        <artifactId>helidon-tracing</artifactId>    
    </dependency>
    <dependency>
        <groupId>io.helidon.webserver.observe</groupId>
        <artifactId>helidon-webserver-observe-tracing</artifactId> 
    </dependency>
</dependencies>
Copied
  • Helidon tracing dependency.
  • Observability dependencies for tracing.

For further processing of the tracing data, different providers are used.

For Jaeger:

<dependency>
    <groupId>io.helidon.tracing.providers</groupId>
    <artifactId>helidon-tracing-providers-jaeger</artifactId>
</dependency>
Copied

For Zipkin:

<dependency>
    <groupId>io.helidon.tracing.providers</groupId>
    <artifactId>helidon-tracing-providers-zipkin</artifactId>
</dependency>
Copied

For OpenTelemetry:

<dependency>
    <groupId>io.helidon.tracing.providers</groupId>
    <artifactId>helidon-tracing-providers-opentelemetry</artifactId>
</dependency>
Copied

For OpenTracing:

<dependency>
    <groupId>io.helidon.tracing.providers</groupId>
    <artifactId>helidon-tracing-providers-opentracing</artifactId>
</dependency>
Copied

Usage

This section explains a few concepts that you need to understand before you get started with tracing.

  • In the context of this document, a service is synonymous with an application.

  • A span is the basic unit of work done within a single service, on a single host. Every span has a name, starting timestamp, and duration. For example, the work done by a REST endpoint is a span. A span is associated to a single service, but its descendants can belong to different services and hosts.

  • A trace contains a collection of spans from one or more services, running on one or more hosts. For example, if you trace a service endpoint that calls another service, then the trace would contain spans from both services. Within a trace, spans are organized as a directed acyclic graph (DAG) and can belong to multiple services, running on multiple hosts.

  • Baggage is a collection of key-value pairs associated with a span.

  • Span context captures data about a span not related to its duration, such as the tracer ID, the span ID, and baggage.

Support for specific tracers is abstracted. Your application can depend on the Helidon abstraction layer and provide a specific tracer implementation as a Java ServiceLoader service. Helidon provides such an implementation for:

  • OpenTracing tracers, either using the GlobalTracer, provider resolver approach, or explicitly using Zipkin tracer

  • OpenTelemetry tracers, either using the global OpenTelemetry instance, or explicitly using Jaeger tracer

Setup WebServer

Configuring Tracer
Tracer tracer = TracerBuilder.create("helidon") 
        .build();

WebServer.builder()
        .addFeature(ObserveFeature.builder()
                            .addObserver(TracingObserver.create(tracer)) 
                            .build())
        .build()
        .start();
Copied
  • Create a Tracer.
  • Add an observability feature using the created Tracer.

Creating Custom Spans

To create a custom span from tracer:

Span span = tracer.spanBuilder("name") 
        .tag("key", "value")
        .start();

try { 
    // do some work
    span.end();
} catch (Throwable t) { 
    span.end(t);
}
Copied
  • Create span from tracer.
  • Do some work and end span.
  • End span with exception.

Handling Baggage

Your application can set and read baggage associated with a Span. The Span.baggage() method returns a WritableBaggage instance.

Further, Helidon also provides read-only access to baggage linked to a SpanContext. For example, HTTP headers can convey trace ID, span ID, and baggage information and Helidon puts such information into a SpanContext. Your code can create a SpanContext from other sources as well. The SpanContext.baggage() method returns a read-only Baggage instance.

The JavaDoc for the types describes how to get and set baggage entries, get all the baggage keys, and check whether a baggage key exists in the baggage.

Responding to Span Lifecycle Events

Applications and libraries can register listeners to be notified at several moments during the lifecycle of every span:

  • Before a new span starts

  • After a new span has started

  • After a span ends

  • After a span is activated (creating a new scope)

  • After a scope is closed

The next sections explain how you can write and add a listener and what it can do. See the SpanListener Javadoc for more information.

Understanding What Listeners Do

A listener cannot affect the lifecycle of a span or scope it is notified about, but it can add tags and events and update the baggage associated with a span. Often a listener does additional work that does not change the span or scope such as logging a message.

When Helidon invokes the listener’s methods it passes proxies for the Span.Builder, Span, and Scope arguments. These proxies limit the access the listener has to the span builder, span, or scope, as summarized in the following table. If a listener method tries to invoke a forbidden operation, the proxy throws a SpanListener.ForbiddenOperationException and Helidon then logs a WARNING message describing the invalid operation invocation.

Summary of Permitted Operations on Proxies Passed to Listeners
Tracing typeChanges allowed
Span.BuilderAdd tags
SpanRetrieve and update baggage, add events, add tags
Scopenone

The following tables list specifically what operations the proxies permit.

MethodPurposeOK?
build()Starts the span.-
end methodsEnds the span.-
get()Starts the span.-
kind(Kind)Sets the "kind" of span (server, client, internal, etc.)-
parent(SpanContext)Sets the parent of the span to be created from the builder.-
start()Starts the span.-
start(Instant)Starts the span.-
tag methodsAdd a tag to the builder before the span is built.
unwrap(Class)Cast the builder to the specified implementation type. †

† Helidon returns the unwrapped object, not a proxy for it.

MethodPurposeOK?
activate()Makes the span "current", returning a Scope.-
addEvent methodsAssociate a string (and optionally other info) with a span.
baggage()Returns the Baggage instance associated with the span.
context()Returns the SpanContext associated with the span.
status(Status)Sets the status of the span.-
any tag methodAdd a tag to the span.
unwrap(Class)Cast the span to the specified implementation type. †

† Helidon returns the unwrapped object, not a proxy to it.

MethodPurposeOK?
close()Close the scope.-
isClosed()Reports whether the scope is closed.
MethodPurposeOK?
asParent(Span.Builder)Sets this context as the parent of a new span builder.
baggage()Returns Baggage instance associated with the span context.
spanId()Returns the span ID.
traceId()Returns the trace ID.

Adding a Listener

Explicitly Registering a Listener on a Tracer

Create a SpanListener instance and invoke the Tracer#register(SpanListener) method to make the listener known to that tracer.

Automatically Registering a Listener on all Tracer Instances

Helidon also uses Java service loading to locate listeners and register them automatically on all Tracer objects. Follow these steps to add a listener service provider.

  1. Implement the SpanListener interface.
  2. Declare your implementation as a service provider:
    1. Create the file META-INF/services/io.helidon.tracing.SpanListener containing a line with the fully-qualified name of your class which implements SpanListener.
    2. If your service has a module-info.java file add the following line to it:
      provides io.helidon.tracing.SpanListener with <your-implementation-class>;
      Copied

The SpanListener interface declares default no-op implementations for all the methods, so your listener can implement only the methods it needs to.

Helidon invokes each listener’s methods in the following order:

Order in which Helidon Invokes Listener Methods
MethodWhen invoked
starting(Span.Builder<?> spanBuilder)Just before a span is started from its builder.
started(Span span)Just after a span has started.
activated(Span span, Scope scope)After a span has been activated, creating a new scope. A given span might never be activated; it depends on the code.
closed(Span span, Scope scope)After a scope has been closed.
ended(Span span)After a span has ended successfully.
ended(Span span, Throwable t)After a span has ended unsuccessfully.

Helidon Spans

Traced spans

The following table lists all spans traced by Helidon components:

componentspan namedescription
web-serverHTTP RequestThe overall span of the Web Server from request initiation until response Note that in Zipkin the name is replaced with jax-rs span name if jax-rs tracing is used.
web-servercontent-readSpan for reading the request entity
web-servercontent-writeSpan for writing the response entity
securitysecurityProcessing of request security
securitysecurity:atnSpan for request authentication
securitysecurity:atzSpan for request authorization
securitysecurity:responseProcessing of response security
securitysecurity:outboundProcessing of outbound security
jax-rsA generated nameSpan for the resource method invocation, name is generated from class and method name
jax-rsjersey-client-callSpan for outbound client call

Some of these spans log to the span. These log events can be (in most cases) configured:

span namelog nameconfigurableenabled by defaultdescription
HTTP Requesthandler.classYESYESEach handler has its class and event logged
securitystatusYESYESLogs either "status: PROCEED" or "status: DENY"
security:atnsecurity.userYESNOThe username of the user if logged in
security:atnsecurity.serviceYESNOThe name of the service if logged in
security:atnstatusYESYESLogs the status of security response (such as SUCCESS)
security:atzstatusYESYESLogs the status of security response (such as SUCCESS)
security:outboundstatusYESYESLogs the status of security response (such as SUCCESS)

There are also tags that are set by Helidon components. These are not configurable.

span nametag namedescription
HTTP Requestcomponentname of the component - helidon-webserver, or jaxrs when using MP
HTTP Requesthttp.methodHTTP method of the request, such as GET, POST
HTTP Requesthttp.status_codeHTTP status code of the response
HTTP Requesthttp.urlThe path of the request (for SE without protocol, host and port)
HTTP RequesterrorIf the request ends in error, this tag is set to true, usually accompanied by logs with details
securitysecurity.idID of the security context created for this request (if security is used)
jersey-client-callhttp.methodHTTP method of the client request
jersey-client-callhttp.status_codeHTTP status code of client response
jersey-client-callhttp.urlFull URL of the request (such as http://localhost:8080/greet)

Configuration

The following configuration should be supported by all tracer implementations (if feasible)

Jaeger tracer configuration.

Type: io.helidon.tracing.Tracer

This is a standalone configuration type, prefix from configuration root: tracing

Configuration options

Optional configuration options
keytypedefault valuedescription
client-cert-pem 

Certificate of client in PEM format.

exporter-timeout

Duration

PT10S

Timeout of exporter requests.

max-export-batch-size

int

512

Maximum Export Batch Size of exporter requests.

max-queue-size

int

2048

Maximum Queue Size of exporter requests.

private-key-pem 

Private key in PEM format.

propagation

PropagationFormat[] (B3, B3_SINGLE, JAEGER, W3C)

JAEGER

Add propagation format to use.

sampler-param

Number

1

The sampler parameter (number).

sampler-type

SamplerType (CONSTANT, RATIO)

CONSTANT

Sampler type.

See <a href="https://www.jaegertracing.io/docs/latest/sampling/#client-sampling-configuration">Sampler types</a>.
schedule-delay

Duration

PT5S

Schedule Delay of exporter requests.

span-processor-type

SpanProcessorType (SIMPLE, BATCH)

batch

Span Processor type used.

trusted-cert-pem 

Trusted certificates in PEM format.

Traced Spans Configuration

Each component and its spans can be configured using Config. The traced configuration has the following layers:

  • TracingConfig - the overall configuration of traced components of Helidon

  • ComponentTracingConfig - a component of Helidon that traces spans (such as web-server, security, jax-rs)

  • SpanTracingConfig - a single traced span within a component (such as security:atn)

  • SpanLogTracingConfig - a single log event on a span (such as security.user in span security:atn)

The components using tracing configuration use the TracingConfigUtil. This uses the io.helidon.common.Context to retrieve current configuration.

Configuration Using Builder

Builder approach, example that disables a single span log event:

Configure tracing using a builder
TracingConfig.builder()
        .addComponent(ComponentTracingConfig.builder("web-server")
                              .addSpan(SpanTracingConfig.builder("HTTP Request")
                                               .addSpanLog(SpanLogTracingConfig.builder("content-write")
                                                                   .enabled(false)
                                                                   .build())
                                               .build())
                              .build())
        .build();
Copied

Configuration using Helidon Config

Tracing configuration can be defined in a config file.

Tracing configuration
tracing:
    components:
      web-server:
        spans:
          - name: "HTTP Request"
            logs:
              - name: "content-write"
                enabled: false
Copied
Use the configuration in web server
Tracer tracer = TracerBuilder.create(config.get("tracing")).build(); 
server.addFeature(ObserveFeature.builder()
                          .addObserver(TracingObserver.create(tracer)) 
                          .build());
Copied
  • Create Tracer using TracerBuilder from configuration.
  • Add the Tracer as an observability feature.

Path-based Configuration in Helidon WebServer

For Web Server we have path-based support for configuring tracing, in addition to the configuration described above.

Configuration of path can use any path string supported by the WebServer. The configuration itself has the same possibilities as traced configuration described above. The path-specific configuration will be merged with global configuration (path is the "newer" configuration, global is the "older")

Configuration in YAML
tracing:
  paths:
    - path: "/favicon.ico"
      enabled: false
    - path: "/metrics"
      enabled: false
    - path: "/health"
      enabled: false
    - path: "/greet"
      components:
        web-server:
          spans:
          - name: "content-read"
            new-name: "read"
            enabled: false
Copied

Renaming top level span using request properties

To have a nicer overview in search pane of a tracer, you can customize the top-level span name using configuration.

Example:

Configuration in YAML
tracing:
  components:
    web-server:
      spans:
      - name: "HTTP Request"
        new-name: "HTTP %1$s %2$s"
Copied

This is supported ONLY for the span named "HTTP Request" on component "web-server".

Parameters provided:

  1. Method - HTTP method
  2. Path - path of the request (such as '/greet')
  3. Query - query of the request (may be null)

Additional Information

WebClient Span Propagation

Span propagation is supported with Helidon WebClient. Tracing propagation is automatic as long as the current span context is available in Helidon Context (which is automatic when running within a WebServer request).

<dependencies>
    <dependency>
        <groupId>io.helidon.webclient</groupId>
        <artifactId>helidon-webclient</artifactId>
    </dependency>
    <dependency>
        <groupId>io.helidon.webclient</groupId>
        <artifactId>helidon-webclient-tracing</artifactId>
    </dependency>
</dependencies>
Copied
Tracing propagation with Helidon WebClient
WebClient client = WebClient.builder()
        .addService(WebClientTracing.create())
        .build();

String response = client.get()
        .uri(uri)
        .requestEntity(String.class);
Copied

Jaeger Tracing

<dependency>
    <groupId>io.helidon.tracing</groupId>
    <artifactId>helidon-tracing-providers-jaeger</artifactId>
</dependency>
Copied

Configuring Jaeger

Jaeger tracer configuration.

Type: io.helidon.tracing.Tracer

This is a standalone configuration type, prefix from configuration root: tracing

Configuration options

Optional configuration options
keytypedefault valuedescription
client-cert-pem 

Certificate of client in PEM format.

exporter-timeout

Duration

PT10S

Timeout of exporter requests.

max-export-batch-size

int

512

Maximum Export Batch Size of exporter requests.

max-queue-size

int

2048

Maximum Queue Size of exporter requests.

private-key-pem 

Private key in PEM format.

propagation

PropagationFormat[] (B3, B3_SINGLE, JAEGER, W3C)

JAEGER

Add propagation format to use.

sampler-param

Number

1

The sampler parameter (number).

sampler-type

SamplerType (CONSTANT, RATIO)

CONSTANT

Sampler type.

See <a href="https://www.jaegertracing.io/docs/latest/sampling/#client-sampling-configuration">Sampler types</a>.
schedule-delay

Duration

PT5S

Schedule Delay of exporter requests.

span-processor-type

SpanProcessorType (SIMPLE, BATCH)

batch

Span Processor type used.

trusted-cert-pem 

Trusted certificates in PEM format.

The following is an example of a Jaeger configuration, specified in the YAML format.

tracing:
    service: "helidon-full-http"
    protocol: "https"
    host: "jaeger"
    port: 14240
Copied

Jaeger Tracing Metrics

As the Jaeger Tracing section describes, you can use Jaeger tracing in your Helidon application.

Zipkin Tracing

<dependency>
    <groupId>io.helidon.tracing.providers</groupId>
    <artifactId>helidon-tracing-providers-zipkin</artifactId>
</dependency>
Copied

Configuring Zipkin

Zipkin tracer configuration

Type: io.opentracing.Tracer

This is a standalone configuration type, prefix from configuration root: tracing

Configuration options

Optional configuration options
keytypedefault valuedescription
api-version

Version (V1, V2)

V2

Version of Zipkin API to use. Defaults to Version#V2.

The following is an example of a Zipkin configuration, specified in the YAML format.

tracing:
  zipkin:
    service: "helidon-service"
    protocol: "https"
    host: "zipkin"
    port: 9987
    api-version: 1
    # this is the default path for API version 2
    path: "/api/v2/spans"
    tags:
      tag1: "tag1-value"
      tag2: "tag2-value"
    boolean-tags:
      tag3: true
      tag4: false
    int-tags:
      tag5: 145
      tag6: 741
Copied

Example of Zipkin trace:

Zipkin example

Reference