Tracing

Overview

Distributed tracing is a critical feature of microservice based applications, since it traces workflow both within a service and across multiple services. This provides insight to sequence and timing data for specific blocks of work, which helps you identify performance and operational issues. Helidon includes support for distributed tracing through its own API, backed by OpenTelemetry. Tracing is integrated with WebServer and Security.

Maven Coordinates

To enable Helidon Tracing, add the following dependency to your project’s pom.xml (see Managing Dependencies).

<dependencies>
    <dependency>
        <groupId>io.helidon.tracing</groupId>
        <artifactId>helidon-tracing</artifactId>1
    </dependency>
    <dependency>
        <groupId>io.helidon.webserver.observe</groupId>
        <artifactId>helidon-webserver-observe-tracing</artifactId>2
    </dependency>
    <dependency>
        <groupId>io.helidon.tracing.providers</groupId>
        <artifactId>helidon-tracing-providers-opentelemetry</artifactId>3
        <scope>runtime</scope>
    </dependency>
</dependencies>

Usage

This section explains a few concepts that you need to understand before you get started with tracing.

• In the context of this document, a service is synonymous with an application.
• A span is the basic unit of work done within a single service, on a single host. Every span has a name, starting timestamp, and duration. For example, the work done by a REST endpoint is a span. A span is associated to a single service, but its descendants can belong to different services and hosts.
• A trace contains a collection of spans from one or more services, running on one or more hosts. For example, if you trace a service endpoint that calls another service, then the trace would contain spans from both services. Within a trace, spans are organized as a directed acyclic graph (DAG) and can belong to multiple services, running on multiple hosts.
• Baggage is a collection of key-value pairs associated with a span.
• Span context captures data about a span not related to its duration, such as the tracer ID, the span ID, and baggage.

Support for specific tracers is abstracted. Your application can depend on the Helidon abstraction layer and provide a specific tracer implementation as a Java ServiceLoader service. Helidon provides OpenTelemetry tracers configured from Helidon tracing settings.

Migrating from Global Tracer APIs

Helidon 27 removes the deprecated static APIs for assigning or retrieving the Helidon global tracer. The OpenTelemetry helper methods which assigned a Helidon global tracer, such as HelidonOpenTelemetry.global(...) and OpenTelemetryTracerProvider.globalTracer(...), are also removed. Application code should obtain the application Tracer from the Helidon service registry, using either declarative injection or Services.get(Tracer.class).

If an application prepares its own tracing runtime, register the manually prepared OpenTelemetry and Tracer instances with Services.set(...) before any application or Helidon component requests either contract from the service registry. Once a registry singleton is resolved, replacing it is not supported.

Some OpenTelemetry instrumentation reads GlobalOpenTelemetry directly. For that interop case, configure telemetry.global: true or tracing.global: true so Helidon publishes the selected application OpenTelemetry instance to the OpenTelemetry global during service-registry ownership startup. This must happen before any other code initializes GlobalOpenTelemetry, because OpenTelemetry allows the JVM-wide global to be assigned only once. If an OpenTelemetry global already exists, Helidon leaves it unchanged and uses the existing global as the application OpenTelemetry instance in the service registry.

WebServer Setup

Configuring Tracer:

Tracer tracer = TracerBuilder.create("helidon")1
        .build();

WebServer.builder()
        .addFeature(ObserveFeature.builder()
                            .addObserver(TracingObserver.create(tracer))2
                            .build())
        .build()
        .start();

Custom Spans

To create a custom span from tracer:

Span span = tracer.spanBuilder("name")1
        .tag("key", "value")
        .start();

try {2
    // do some work
    span.end();
} catch (Throwable t) {3
    span.end(t);
}

Handling Baggage

Your application can set and read baggage associated with a Span. The Span.baggage() method returns a WritableBaggage instance.

Further, Helidon also provides read-only access to baggage linked to a SpanContext. For example, HTTP headers can convey trace ID, span ID, and baggage information and Helidon puts such information into a SpanContext. Your code can create a SpanContext from other sources as well. The SpanContext.baggage() method returns a read-only Baggage instance.

The Javadoc for the types describes how to get and set baggage entries, get all the baggage keys, and check whether a baggage key exists in the baggage.

Span Lifecycle

Events

Applications and libraries can register listeners to be notified at several moments during the lifecycle of every Helidon span:

• Before a new span starts
• After a new span has started
• After a span ends
• After a span is activated (creating a new scope)
• After a scope is closed

The next sections explain how you can write and add a listener and what it can do. See the SpanListener Javadoc for more information.

Listeners

A listener cannot affect the lifecycle of a span or scope it is notified about, but it can add tags and events and update the baggage associated with a span. Often a listener does additional work that does not change the span or scope such as logging a message.

When Helidon invokes the listener’s methods it passes proxies for the Span.Builder, Span, and Scope arguments. These proxies limit the access the listener has to the span builder, span, or scope, as summarized in the following table. If a listener method tries to invoke a forbidden operation, the proxy throws a SpanListener.ForbiddenOperationException and Helidon then logs a WARNING message describing the invalid operation invocation.

Summary of Permitted Operations on Proxies Passed to Listeners

The following tables list specifically what operations the proxies permit.

io.helidon.tracing.Span.Builder Operations

io.helidon.tracing.Span Operations

io.helidon.tracing.Scope Operations

io.helidon.tracing.SpanContext Operations

Adding a Listener

Explicitly Registering a Listener on a Tracer

Create a SpanListener instance and invoke the Tracer#register(SpanListener) method to make the listener known to that tracer.

Automatically Registering a Listener on all Tracer Instances

Helidon also uses Java service loading to locate listeners and register them automatically on all Tracer objects. Follow these steps to add a listener service provider.

1. Implement the SpanListener interface.
2. Declare your implementation as a service provider:
   1. Create the file META-INF/services/io.helidon.tracing.SpanListener containing a line with the fully-qualified name of your class which implements SpanListener.
   2. If your service has a module-info.java file add the following line to it:

      provides io.helidon.tracing.SpanListener with <your-implementation-class>;
      

The SpanListener interface declares default no-op implementations for all the methods, so your listener can implement only the methods it needs to.

Helidon invokes each listener’s methods in the following order:

Order in which Helidon Invokes Listener Methods

OpenTelemetry Callbacks

To use lifecycle callbacks, applications should normally work with the Helidon Tracer, Span.Builder, Span, and Scope types which automatically call back to each registered SpanListener.

In some cases application code might want to use a reference to an OpenTelemetry Tracer or Span rather than a reference to the Helidon counterpart but still want to respond to lifecycle events as the OpenTelemetry object goes through its lifecycle.

The HelidonOpenTelemetry type provides several methods which enable callbacks for OpenTelemetry objects, as summarized in the following table.

Enabling OpenTelemetry Objects for SpanListener Support

An OpenTelemetry object returned from a method on a callback-enabled object is itself callback-enabled automatically. Specifically:

• SpanBuilder returned from Tracer#spanBuilder(String).
• Span returned from SpanBuilder#startSpan.
• Scope returned from Span#makeCurrent.

Each callback-enabled object is a new instance of a Helidon object which implements both the indicated OpenTelemetry interface and the Helidon Wrapper interface. These Helidon objects do not themselves implement other OpenTelemetry interfaces. To do type checks and casts on callback-enabled objects, invoke the unwrap(Class<?>) on a callback-enabled object as shown in the following example.

// Note that callbackEnabledSpan implements OpenTelemetry Span.
io.opentelemetry.api.trace.Span nativeOtelSpan = callbackEnabledSpan.unwrap(io.opentelemetry.api.trace.Span.class);
if (nativeOtelSpan instanceof ReadableSpan readableSpan) {
    // Work with the span as a ReadableSpan
}

Remember that operations on the nativeOtelSpan variable do not notify span listeners of lifecycle changes.

Configuration options

Helidon Spans

The following table lists all spans traced by Helidon components:

Some of these spans log to the span. These log events can be (in most cases) configured:

There are also tags that are set by Helidon components. These are not configurable.

Configuration

Each component and its spans can be configured using Config. The traced configuration has the following layers:

• TracingConfig - the overall configuration of traced components of Helidon
• ComponentTracingConfig - a component of Helidon that traces spans (such as web-server and security)
• SpanTracingConfig - a single traced span within a component (such as security:atn)
• SpanLogTracingConfig - a single log event on a span (such as security.user in span security:atn)

The components using tracing configuration use the TracingConfigUtil. This uses the io.helidon.common.Context to retrieve current configuration.

Configuration Using Builder

Builder approach, example that disables a single span log event:

Configure tracing using a builder:

TracingConfig.builder()
    .addComponent(ComponentTracingConfig.builder("web-server")
        .addSpan(SpanTracingConfig.builder("HTTP Request")
            .addSpanLog(SpanLogTracingConfig.builder("content-write")
                 .enabled(false)
                 .build())
            .build())
        .build())
    .build();

Configuration using Helidon Config

Tracing configuration can be defined in a config file.

Tracing configuration:

tracing:
    components:
      web-server:
        spans:
          - name: "HTTP Request"
            logs:
              - name: "content-write"
                enabled: false

Use the configuration in web server:

Tracer tracer = TracerBuilder.create(config.get("tracing")).build();1
server.addFeature(ObserveFeature.builder()
                          .addObserver(TracingObserver.create(tracer))2
                          .build());

Path-based Configuration in Helidon WebServer

For Web Server we have path-based support for configuring tracing, in addition to the configuration described above.

Configuration of path can use any path string supported by the WebServer. The configuration itself has the same possibilities as traced configuration described above. The path-specific configuration will be merged with global configuration (path is the "newer" configuration, global is the "older")

Configuration in YAML:

tracing:
  paths:
    - path: "/favicon.ico"
      enabled: false
    - path: "/metrics"
      enabled: false
    - path: "/health"
      enabled: false
    - path: "/greet"
      methods: ["GET"]
      components:
        web-server:
          spans:
          - name: "content-read"
            new-name: "read"
            enabled: false

Configured method names are matched exactly.

Renaming top level span using request properties

To have a nicer overview in search pane of a tracer, you can customize the top-level span name using configuration.

Example:

Configuration in YAML:

tracing:
  components:
    web-server:
      spans:
      - name: "HTTP Request"
        new-name: "HTTP %1$s %2$s"

This is supported ONLY for the span named "HTTP Request" on component "web-server".

Parameters provided:

1. Method - HTTP method
2. Path - path of the request (such as /greet)
3. Query - query of the request (maybe null)

WebClient Propagation

Span propagation is supported with Helidon WebClient. Tracing propagation is automatic as long as the current span context is available in Helidon Context (which is automatic when running within a WebServer request).

<dependencies>
  <dependency>
    <groupId>io.helidon.webclient</groupId>
    <artifactId>helidon-webclient</artifactId>
  </dependency>
  <dependency>
    <groupId>io.helidon.webclient</groupId>
    <artifactId>helidon-webclient-tracing</artifactId>
  </dependency>
</dependencies>

Tracing propagation with Helidon WebClient:

WebClient client = WebClient.builder()
        .addService(WebClientTracing.create())
        .build();

String response = client.get()
        .uri(uri)
        .requestEntity(String.class);

OpenTelemetry Tracing

Helidon supports configuration of OpenTelemetry and OpenTelemetry tracing in two primary ways: using tracing or using telemetry. This page describes support for controlling OpenTelemetry tracing using the tracing config section and OpenTelemetryTracerConfig builder. Users typically adopt this approach to ease migration from older tracing configuration to OpenTelemetry.

That said, Helidon’s support for OpenTelemetry using tracing does not afford as much control as do the Helidon telemetry settings. For example, using OpenTelemetry tracing config you can choose either the OTLP gRPC span exporter or the OTLP HTTP one; additional span exporters are available only using the telemetry settings.

The telemetry doc page describes how to use the Helidon telemetry config section and the related builder to exert more control over OpenTelemetry and OpenTelemetry tracing behavior.

Dependency for OpenTelemetry support using tracing:

<dependency>
  <groupId>io.helidon.tracing.providers</groupId>
  <artifactId>helidon-tracing-providers-opentelemetry</artifactId>
</dependency>

Configuration options

Example Helidon configuration for OpenTelemetry tracing:

tracing:
  service: helidon-otel-tracing-example1
  global: true2
  int-tags:
    example: 13
  tags:
    direction: north4

Set tracing.registered to false only when the tracing section should not contribute the application-wide OpenTelemetry and Tracer from the Helidon service registry. OpenTelemetry globals are JVM-wide and can be assigned only once, so any tracing.global: true publication must happen before any Java agent, OpenTelemetry autoconfiguration, or application code initializes GlobalOpenTelemetry. If an OpenTelemetry global already exists, Helidon leaves it unchanged and uses the existing global as the application OpenTelemetry instance in the service registry.

By default, Helidon tracing support for OpenTelemetry uses OpenTelemetry’s OTLP gRPC exporter. Alternatively, you can choose to use OpenTelemetry’s HTTP exporter using protobuf by setting exporter-type to http/proto. To use other exporters OpenTelemetry offers, use the Helidon telemetry configuration instead of tracing.

Reference

• OpenTelemetry API