Contents

Overview

Maven Coordinates

To enable MicroProfile Telemetry either add a dependency on the helidon-microprofile bundle or add the following dependency to your project’s pom.xml (see Managing Dependencies).

<dependency>
    <groupId>io.helidon.microprofile.telemetry</groupId>
    <artifactId>helidon-microprofile-telemetry</artifactId>
</dependency>
Copied

Usage

OpenTelemetry comprises a collection of APIs, SDKs, integration tools, and other software components intended to facilitate the generation and control of telemetry data, including traces, metrics, and logs. In an environment where distributed tracing is enabled via OpenTelemetry (which combines OpenTracing and OpenCensus), this specification establishes the necessary behaviors for MicroProfile applications to participate seamlessly.

MicroProfile Telemetry 1.0 allows for the exportation of the data it collects to Jaeger or Zipkin and to other systems using a variety of exporters.

In a distributed tracing system, traces are used to capture a series of requests and are composed of multiple spans that represent individual operations within those requests. Each span includes a name, timestamps, and metadata that provide insights into the corresponding operation.

Context is included in each span to identify the specific request that it belongs to. This context information is crucial for tracking requests across various components in a distributed system, enabling developers to trace a single request as it traverses through multiple services.

Finally, exporters are responsible for transmitting the collected trace data to a backend service for monitoring and visualization. This enables developers to gain a comprehensive understanding of the system’s behavior and detect any issues or bottlenecks that may arise.

General understanding of OpenTelemetry Tracing

There are two ways to work with Telemetry, using:

  • Automatic Instrumentation

  • Manual Instrumentation

For Automatic Instrumentation, OpenTelemetry provides a JavaAgent. The Tracing API allows for the automatic participation in distributed tracing of Jakarta RESTful Web Services (both server and client) as well as MicroProfile REST Clients, without requiring any modifications to the code. This is achieved through automatic instrumentation.

For Manual Instrumentation, there is a set of annotations and access to OpenTelemetry API.

@WithSpan - By adding this annotation to a method in any Jakarta CDI aware bean, a new span will be created and any necessary connections to the current Trace context will be established. Additionally, the SpanAttribute annotation can be used to mark method parameters that should be included in the Trace.

Helidon provides full access to OpenTelemetry Tracing API:

  • io.opentelemetry.api.OpenTelemetry

  • io.opentelemetry.api.trace.Tracer

  • io.opentelemetry.api.trace.Span

  • io.opentelemetry.api.baggage.Baggage

Accessing and using these objects can be done as follows. For span:

Span sample
@ApplicationScoped
class HelidonBean {

    @WithSpan 
    void doSomethingWithinSpan() {
        // do something here
    }

    @WithSpan("name") 
    void complexSpan(@SpanAttribute(value = "arg") String arg) {
        // do something here
    }
}
Copied
  • Simple @WithSpan annotation usage.
  • Additional attributes can be set on a method.

Working With Tracers

You can inject OpenTelemetry Tracer using the regular @Inject annotation and use SpanBuilder to manually create, star and stop spans.

SpanBuilder usage
@Path("/")
public class HelidonEndpoint {

    @Inject
    Tracer tracer; 

    @GET
    @Path("/span")
    public Response span() {
        Span span = tracer.spanBuilder("new") 
                .setSpanKind(SpanKind.CLIENT)
                .setAttribute("someAttribute", "someValue")
                .startSpan();

        span.end();

        return Response.ok().build();
    }
}
Copied
  • Inject Tracer.
  • Use Tracer.spanBuilder to create and start new Span.

Helidon Microprofile Telemetry is integrated with Helidon Tracing API. This means that both APIs can be mixed, and all parent hierarchies will be kept. In the case below, @WithSpan annotated method is mixed with manually created io.helidon.tracing.Span:

Inject Helidon Tracer
private io.helidon.tracing.Tracer helidonTracerInjected;

@Inject
GreetResource(io.helidon.tracing.Tracer helidonTracerInjected) {
    this.helidonTracerInjected = helidonTracerInjected; 
}

@GET
@Path("mixed_injected")
@Produces(MediaType.APPLICATION_JSON)
@WithSpan("mixed_parent_injected")
public GreetingMessage mixedSpanInjected() {
    io.helidon.tracing.Span mixedSpan = helidonTracerInjected.spanBuilder("mixed_injected") 
            .kind(io.helidon.tracing.Span.Kind.SERVER)
            .tag("attribute", "value")
            .start();
    mixedSpan.end();

    return new GreetingMessage("Mixed Span Injected" + mixedSpan);
}
Copied
  • Inject io.helidon.tracing.Tracer.
  • Use the injected tracer to create io.helidon.tracing.Span using the spanBuilder() method.

The span is then started and ended manually. Span parent relations will be preserved. This means that span named "mixed_injected" with have parent span named "mixed_parent_injected", which will have parent span named "mixed_injected".

Another option is to use the Global Tracer:

Obtain the Global tracer
@GET
@Path("mixed")
@Produces(MediaType.APPLICATION_JSON)
@WithSpan("mixed_parent")
public GreetingMessage mixedSpan() {
    io.helidon.tracing.Tracer helidonTracer = io.helidon.tracing.Tracer.global(); 
    io.helidon.tracing.Span mixedSpan = helidonTracer.spanBuilder("mixed") 
            .kind(io.helidon.tracing.Span.Kind.SERVER)
            .tag("attribute", "value")
            .start();
    mixedSpan.end();

    return new GreetingMessage("Mixed Span" + mixedSpan);
}
Copied
  • Obtain tracer using the io.helidon.tracing.Tracer.global() method;
  • Use the created tracer to create a span.

The span is then started and ended manually. Span parent relations will be preserved.

Working With Spans

To obtain the current span, it can be injected by CDI. The current span can also be obtained using the static method Span.current().

Inject the current span
@Path("/")
public class HelidonEndpoint {
    @Inject
    Span span; 

    @GET
    @Path("/current")
    public Response currentSpan() {
        return Response.ok(span).build(); 
    }

    @GET
    @Path("/current/static")
    public Response currentSpanStatic() {
        return Response.ok(Span.current()).build(); 
    }
}
Copied
  • Inject the current span.
  • Use the injected span.
  • Use Span.current() to access the current span.

Working With Baggage

The same functionality is available for the Baggage API:

Inject the current baggage
@Path("/")
public class HelidonEndpoint {
    @Inject
    Baggage baggage; 

    @GET
    @Path("/current")
    public Response currentBaggage() {
        return Response.ok(baggage.getEntryValue("baggageKey")).build(); 
    }

    @GET
    @Path("/current/static")
    public Response currentBaggageStatic() {
        return Response.ok(Baggage.current().getEntryValue("baggageKey")).build(); 
    }
}
Copied
  • Inject the current baggage.
  • Use the injected baggage.
  • Use Baggage.current() to access the current baggage.

Responding to Span Lifecycle Events

Applications and libraries can register listeners to be notified at several moments during the lifecycle of every span:

  • Before a new span starts

  • After a new span has started

  • After a span ends

  • After a span is activated (creating a new scope)

  • After a scope is closed

The next sections explain how you can write and add a listener and what it can do. See the SpanListener Javadoc for more information.

Understanding What Listeners Do

A listener cannot affect the lifecycle of a span or scope it is notified about, but it can add tags and events and update the baggage associated with a span. Often a listener does additional work that does not change the span or scope such as logging a message.

When Helidon invokes the listener’s methods it passes proxies for the Span.Builder, Span, and Scope arguments. These proxies limit the access the listener has to the span builder, span, or scope, as summarized in the following table. If a listener method tries to invoke a forbidden operation, the proxy throws a SpanListener.ForbiddenOperationException and Helidon then logs a WARNING message describing the invalid operation invocation.

Summary of Permitted Operations on Proxies Passed to Listeners
Tracing typeChanges allowed
Span.BuilderAdd tags
SpanRetrieve and update baggage, add events, add tags
Scopenone

The following tables list specifically what operations the proxies permit.

MethodPurposeOK?
build()Starts the span.-
end methodsEnds the span.-
get()Starts the span.-
kind(Kind)Sets the "kind" of span (server, client, internal, etc.)-
parent(SpanContext)Sets the parent of the span to be created from the builder.-
start()Starts the span.-
start(Instant)Starts the span.-
tag methodsAdd a tag to the builder before the span is built.
unwrap(Class)Cast the builder to the specified implementation type. †

† Helidon returns the unwrapped object, not a proxy for it.

MethodPurposeOK?
activate()Makes the span "current", returning a Scope.-
addEvent methodsAssociate a string (and optionally other info) with a span.
baggage()Returns the Baggage instance associated with the span.
context()Returns the SpanContext associated with the span.
status(Status)Sets the status of the span.-
any tag methodAdd a tag to the span.
unwrap(Class)Cast the span to the specified implementation type. †

† Helidon returns the unwrapped object, not a proxy to it.

MethodPurposeOK?
close()Close the scope.-
isClosed()Reports whether the scope is closed.
MethodPurposeOK?
asParent(Span.Builder)Sets this context as the parent of a new span builder.
baggage()Returns Baggage instance associated with the span context.
spanId()Returns the span ID.
traceId()Returns the trace ID.

Adding a Listener

Explicitly Registering a Listener on a Tracer

Create a SpanListener instance and invoke the Tracer#register(SpanListener) method to make the listener known to that tracer.

Automatically Registering a Listener on all Tracer Instances

Helidon also uses Java service loading to locate listeners and register them automatically on all Tracer objects. Follow these steps to add a listener service provider.

  1. Implement the SpanListener interface.
  2. Declare your implementation as a service provider:
    1. Create the file META-INF/services/io.helidon.tracing.SpanListener containing a line with the fully-qualified name of your class which implements SpanListener.
    2. If your service has a module-info.java file add the following line to it:
      provides io.helidon.tracing.SpanListener with <your-implementation-class>;
      Copied

The SpanListener interface declares default no-op implementations for all the methods, so your listener can implement only the methods it needs to.

Helidon invokes each listener’s methods in the following order:

Order in which Helidon Invokes Listener Methods
MethodWhen invoked
starting(Span.Builder<?> spanBuilder)Just before a span is started from its builder.
started(Span span)Just after a span has started.
activated(Span span, Scope scope)After a span has been activated, creating a new scope. A given span might never be activated; it depends on the code.
closed(Span span, Scope scope)After a scope has been closed.
ended(Span span)After a span has ended successfully.
ended(Span span, Throwable t)After a span has ended unsuccessfully.

Configuration

MicroProfile Telemetry is not activated by default. To activate this feature, you need to specify the configuration otel.sdk.disabled=false in one of the MicroProfile Config or other config sources.

To configure OpenTelemetry, MicroProfile Config must be used, and the configuration properties outlined in the following sections must be followed:

Please consult with the links above for all configurations' properties usage.

The property should be declared in microprofile-config.properties file to be processed correctly.

OpenTelemetry Java Agent

The OpenTelemetry Java Agent may influence the work of MicroProfile Telemetry, on how the objects are created and configured. Helidon will do "best effort" to detect the use of the agent. But if there is a decision to run the Helidon app with the agent, a configuration property should be set:

otel.agent.present=true

This way, Helidon will explicitly get all the configuration and objects from the Agent, thus allowing correct span hierarchy settings.

Examples

This guide demonstrates how to incorporate MicroProfile Telemetry into Helidon and provides illustrations of how to view traces. Jaeger is employed in all the examples, and the Jaeger UI is used to view the traces.

Set Up Jaeger

For example, Jaeger will be used for gathering of the tracing information.

Run Jaeger in a docker container.
docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.50
Copied

All the tracing information gathered from the examples runs is accessible from the browser in the Jaeger UI under http://localhost:16686/

Enable MicroProfile Telemetry in Helidon Application

Together with Helidon Telemetry dependency, an OpenTelemetry Exporter dependency should be added to project’s pom.xml file.

<dependencies>
    <dependency>
        <groupId>io.helidon.microprofile.telemetry</groupId>
        <artifactId>helidon-microprofile-telemetry</artifactId> 
    </dependency>
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-exporter-jaeger</artifactId>  
    </dependency>
</dependencies>
Copied
  • Helidon Telemetry dependency.
  • OpenTelemetry Jaeger exporter.

Add these lines to META-INF/microprofile-config.properties:

MicroProfile Telemetry properties
otel.sdk.disabled=false     
otel.traces.exporter=jaeger 
otel.exporter.name=greeting-service 
Copied
  • Enable MicroProfile Telemetry.
  • Set exporter to Jaeger.
  • Name of our service.

Here we enable MicroProfile Telemetry, set tracer to "jaeger" and give a name, which will be used to identify our service in the tracer.

Note

For this example, you will use Jaeger to manage data tracing. If you prefer to use Zipkin, please set otel.traces.exporter property to "zipkin". For more information using about Zipkin, see https://zipkin.io/. Also, a corresponding Maven dependency for the exporter should be added:

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-zipkin</artifactId>
</dependency>

Tracing at Method Level

To create simple services, use @WithSpan and Tracer to create span and let MicroProfile OpenTelemetry handle them.

@Path("/greet")
public class GreetResource {

    @GET
    @WithSpan("default") 
    public String getDefaultMessage() {
        return "Hello World";
    }
}
Copied
  • Use of @WithSpan with name "default".

Now let’s call the Greeting endpoint:

curl localhost:8080/greet
Hello World
Copied

Next, launch the Jaeger UI at http://localhost:16686/. The expected output is:

Greeting service tracing output
Custom method
@Inject
private Tracer tracer; 

@GET
@Path("custom")
@Produces(MediaType.APPLICATION_JSON)
@WithSpan 
public JsonObject useCustomSpan() {
    Span span = tracer.spanBuilder("custom") 
            .setSpanKind(SpanKind.INTERNAL)
            .setAttribute("attribute", "value")
            .startSpan();
    span.end(); 

    return Json.createObjectBuilder()
            .add("Custom Span", span.toString())
            .build();
}
Copied
  • Inject OpenTelemetry Tracer.
  • Create a span around the method useCustomSpan().
  • Create a custom INTERNAL span and start it.
  • End the custom span.

Let us call the custom endpoint:

curl localhost:8080/greeting/custom
Copied

Again you can launch the Jaeger UI at http://localhost:16686/. The expected output is:

Custom span usage

Now let us use multiple services calls. In the example below our main service will call the secondary services. Each method in each service will be annotated with @WithSpan annotation.

Outbound method
@Uri("http://localhost:8081/secondary")
private WebTarget target; 

@GET
@Path("/outbound")
@WithSpan("outbound") 
public String outbound() {
    return target.request().accept(MediaType.TEXT_PLAIN).get(String.class); 
}
Copied
  • Inject WebTarget pointing to Secondary service.
  • Wrap method using WithSpan.
  • Call the secondary service.

The secondary service is basic; it has only one method, which is also annotated with @WithSpan.

Secondary service
@GET
@WithSpan 
public String getSecondaryMessage() {
    return "Secondary"; 
}
Copied
  • Wrap method in a span.
  • Return a string.

Let us call the Outbound endpoint:

curl localhost:8080/greet/outbound
Secondary
Copied

The greeting-service call secondary-service. Each service will create spans with corresponding names, and a service class hierarchy will be created.

Launch the Jaeger UI at http://localhost:16686/ to see the expected output (shown below).

Secondary service outbound call

This example is available at the Helidon official GitHub repository.

Reference