This section explains a few concepts that you need to understand before you get started with tracing. In the context of this document, a service is synonymous with an application. A span is the basic unit of work done within a single service, on a single host. Every span has a name, starting timestamp, and duration. For example, the work done by a REST endpoint is a span. A span is associated to a single service, but its descendants can belong to different services and hosts. A trace contains a collection of spans from one or more services, running on one or more hosts. For example, if you trace a service endpoint that calls another service, then the trace would contain spans from both services. Within a trace, spans are organized as a directed acyclic graph (DAG) and can belong to multiple services, running on multiple hosts. Spans are automatically created by Helidon as needed during execution of the REST request.

Use the Helidon SE Maven archetype to create a simple project that can be used for the examples in this guide.

mvn -U archetype:generate -DinteractiveMode=false \
    -DarchetypeGroupId=io.helidon.archetypes \
    -DarchetypeArtifactId=helidon-quickstart-se \
    -DarchetypeVersion=4.4.1 \
    -DgroupId=io.helidon.examples \
    -DartifactId=helidon-quickstart-se \
    -Dpackage=io.helidon.examples.quickstart.se

cd helidon-quickstart-se

First, run the Jaeger backend. Helidon communicates with this backend at runtime.

docker run -d --name jaeger \ 
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.50

• Run the Jaeger docker image.

Update the pom.xml file and add the following OpenTelemetry dependency to the <dependencies> section (not <dependencyManagement>). This will enable Helidon to use OpenTelemetry at the default host and port, localhost:4317.

<dependencies>
     <dependency>
         <groupId>io.helidon.tracing</groupId>
         <artifactId>helidon-tracing</artifactId>   
     </dependency>
     <dependency>
         <groupId>io.helidon.webserver.observe</groupId>
         <artifactId>helidon-webserver-observe-tracing</artifactId> 
         <scope>runtime</scope>
     </dependency>
     <dependency>
         <groupId>io.helidon.tracing.providers</groupId>
         <artifactId>helidon-tracing-providers-opentelemetry</artifactId>  
         <scope>runtime</scope>
     </dependency>
</dependencies>

• Helidon Tracing dependencies.
• Observability features for tracing.
• OpenTelemetry tracing provider.

Helidon offers several tracing providers: OpenTelemetry, Zipkin, and Jaeger (deprecated). All spans sent by Helidon to the backend need to be associated with a service, assigned by the tracing.service setting in the example below.

tracing:
  service: helidon-se-1
  tags:
    env: development
  enabled: true
  sampler-type: "const"
  sampler-param: 1
  log-spans: true
  propagation: b3

Tracing is part of Helidon’s observability support. By default, Helidon discovers any observability feature on the classpath and activates it automatically. In particular for tracing, Helidon adds a trace each time a client accesses a service endpoint. You can see these traces using the Jaeger UI once you build, run, and access your application without changing your application’s Java code.

Build and Access QuickStart

mvn clean package
java -jar target/helidon-quickstart-se.jar

curl http://localhost:8080/greet

The Jaeger backend provides a web-based UI at http://localhost:16686 where you can see a visual representation of the traces and spans within them.

1. From the Service drop list select helidon-se-1. This name corresponds to the tracing.service setting you assigned in the application.yaml config file.
2. Click on the UI Find Traces button. Notice that you can change the look-back time to restrict the trace list. You will see a trace for each curl command you ran to access the application.

List of traces

Click on a trace to see the trace detail page (shown below) which shows the spans within the trace. You can clearly see the root span (HTTP Request) and the single child span (content-write) along with the time over which each span was active.

Trace detail page

You can examine span details by clicking on the span row. Refer to the image below which shows the span details including timing information. You can see times for each space relative to the root span.

Span detail page

Your application can use the Helidon tracing API to create custom spans. The following code replaces the generated getDefaultMessageHandler method to add a custom span around the code which prepares the default greeting response. The new custom span’s parent span is set to the one which Helidon automatically creates for the REST endpoint.

private void getDefaultMessageHandler(ServerRequest request,
                                      ServerResponse response) {
    var spanBuilder = Tracer.global().spanBuilder("secondchildSpan"); 
    request.context().get(SpanContext.class).ifPresent(sc -> sc.asParent(spanBuilder)); 
    var span = spanBuilder.start(); 

    try (Scope scope = span.activate()) { 
        sendResponse(response, "World");
        span.end(); 
    } catch (Throwable t) {
        span.end(t);    
    }
}

• Create a new Span using the global tracer.
• Set the parent of the new span to the span from the Request if available.
• Start the span.
• Make the new span the current span, returning a Scope which is auto-closed.
• End the span normally after the response is sent.
• End the span with an exception if one was thrown.

mvn package
java -jar target/helidon-quickstart-se.jar

curl http://localhost:8080/greet

{
  "message": "Hello World!"
}

Return to the main Jaeger UI screen and click Find Traces again. The new display contains an additional trace, displayed first, for the most recent curl you ran.

Expanded trace list

Notice that the top trace has three spans, not two as with the earlier trace. Click on the trace to see the trace details.

Trace details with custom span

Note the row for mychildSpan--the custom span created by the added code.

Helidon automatically traces across services if the services propagate span information. This means a single trace can include spans from multiple services and hosts. Helidon uses a SpanContext to propagate tracing information across process boundaries. When you make client API calls, Helidon will internally call OpenTelemetry APIs or OpenTracing APIs to propagate the SpanContext. There is nothing you need to do in your application to make this work.

To demonstrate distributed tracing, create a second project where the server listens to on port 8081. Create a new directory to hold this new project, then do the following steps, similar to what you did at the start of this guide:

mvn -U archetype:generate -DinteractiveMode=false \
    -DarchetypeGroupId=io.helidon.archetypes \
    -DarchetypeArtifactId=helidon-quickstart-se \
    -DarchetypeVersion=4.4.1 \
    -DgroupId=io.helidon.examples \
    -DartifactId=helidon-quickstart-se-2 \
    -Dpackage=io.helidon.examples.quickstart.se

cd helidon-quickstart-se-2

<dependencies>
     <dependency>
         <groupId>io.helidon.tracing</groupId>
         <artifactId>helidon-tracing</artifactId>   
     </dependency>
     <dependency>
         <groupId>io.helidon.webserver.observe</groupId>
         <artifactId>helidon-webserver-observe-tracing</artifactId> 
         <scope>runtime</scope>
     </dependency>
     <dependency>
         <groupId>io.helidon.tracing.providers</groupId>
         <artifactId>helidon-tracing-providers-opentelemetry</artifactId>  
         <scope>runtime</scope>
     </dependency>
</dependencies>

• Helidon Tracing API.
• Observability features for tracing.
• OpenTelemetry tracing provider.

app:
  greeting: "Hello From SE-2"

tracing:
  service: helidon-se-2
  tags:
    env: development
  enabled: true
  sampler-type: "const"
  sampler-param: 1
  log-spans: true
  propagation: b3

server:
  port: 8081
  host: 0.0.0.0

private void getDefaultMessageHandler(ServerRequest request,
                                      ServerResponse response) {

    var spanBuilder = Tracer.global().spanBuilder("getDefaultMessageHandler");
    request.context().get(SpanContext.class).ifPresent(spanBuilder::parent);
    Span span = spanBuilder.start();

    try (Scope scope = span.activate()) {
        sendResponse(response, "World");
        span.end();
    } catch (Throwable t) {
        span.end(t);
    }
}

Build the application, skipping unit tests; the unit tests check for the default greeting response which is now different in the updated config. Then run the application.

mvn package -DskipTests=true
java -jar target/helidon-quickstart-se-2.jar

curl http://localhost:8081/greet

{
  "message": "Hello From SE-2 World!"
}

Once you have validated that the second service is running correctly, you need to modify the original application to call it.

<dependencies>
    <dependency>
        <groupId>io.helidon.webclient</groupId>
        <artifactId>helidon-webclient</artifactId>
    </dependency>
    <dependency>
        <groupId>io.helidon.webclient</groupId>
        <artifactId>helidon-webclient-api</artifactId>
    </dependency>
    <dependency>
        <groupId>io.helidon.webclient</groupId>
        <artifactId>helidon-webclient-tracing</artifactId>
    </dependency>
    <dependency>
        <groupId>io.helidon.webclient</groupId>
        <artifactId>helidon-webclient-http1</artifactId>
        <scope>runtime</scope>
    </dependency>
</dependencies>

Make the following changes to the GreetFeature class.

1. Add a WebClient field.
   Add a private instance field (before the constructors)

   private WebClient webClient;

   content_copy Copied
2. Add code to initialize the WebClient field.
   Add the following code to the GreetService(Config) constructor

   webClient = WebClient.builder()
           .baseUri("http://localhost:8081")
           .addService(WebClientTracing.create())
           .build();

   content_copy Copied
3. Add a routing rule for the new endpoint /outbound.
   Add the following line in the routing method as the first .get invocation in the method

   .get("/outbound", this::outboundMessageHandler);

   content_copy Copied
4. Add a method to handle requests to /outbound.
   Add the following method

   private void outboundMessageHandler(ServerRequest request,
                                       ServerResponse response) {
       var spanBuilder = Tracer.global().spanBuilder("outboundMessageHandler");
       request.context().get(SpanContext.class).ifPresent(spanBuilder::parent);
       var span = spanBuilder.start();
   
       try (Scope scope = span.activate()) {
           ClientResponseTyped<JsonObject> remoteResult = webClient.get()
                   .path("/greet")
                   .accept(MediaTypes.APPLICATION_JSON)
                   .request(JsonObject.class);
   
           response.status(remoteResult.status()).send(remoteResult.entity());
           span.end();
       } catch (Exception e) {
           response.status(Status.INTERNAL_SERVER_ERROR_500).send();
           span.end(e);
       }
   }

   content_copy Copied

Stop the application if it is still running, rebuild and run it, then invoke the endpoint and check the response.

mvn clean package
java -jar target/helidon-quickstart-se.jar
curl -i http://localhost:8080/greet/outbound 

• The request goes to the service on 8080, which then invokes the service at 8081 to get the greeting.

{
  "message": "Hello From SE-2 World!" 
}

• Notice the greeting came from the second service.

Refresh the Jaeger UI trace listing page and notice that there is a trace across two services. Click on that trace to see its details.

Tracing across multiple services detail view

Note several things about the display:

1. The top-level span helidon-se-1 HTTP Request includes all the work across both services.
2. helidon-se-1 outboundMessageHandler is the custom span you added to the first service /outbound endpoint code.
3. helidon-se-1 GET-http://localhost:8080/greet captures the work the WebClient is doing in sending a request to the second service. Helidon adds these spans automatically to each outbound WebClient request.
4. helidon-se-2 HTTP Request represents the arrival of the request sent by the first service’s WebClient at the second service’s /greet endpoint.
5. helidon-se-2 getDefaultMessageHandler is the custom span you added to the second service /greet endpoint code.

You can now stop your second service, it is no longer used in this guide.

apiVersion: v1
kind: Service
metadata:
  name: jaeger
spec:
  ports:
    - port: 16686
      protocol: TCP
  selector:
    app: jaeger
---
kind: Pod
apiVersion: v1
metadata:
  name: jaeger
  labels:
    app: jaeger
spec:
  containers:
    - name: jaeger
      image: jaegertracing/all-in-one
      imagePullPolicy: IfNotPresent
      ports:
        - containerPort: 16686

kubectl apply -f ./jaeger.yaml

kubectl expose pod  jaeger --name=jaeger-external --port=16687 --target-port=16686 --type=LoadBalancer

Navigate to http://localhost:16687/jaeger to validate that you can access Jaeger running in Kubernetes. It may take a few seconds before it is ready.

kind: Service
apiVersion: v1
metadata:
  name: helidon-tracing 
  labels:
    app: helidon-tracing
spec:
  type: NodePort
  selector:
    app: helidon-tracing
  ports:
    - port: 8080
      targetPort: 8080
      name: http
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: helidon-tracing
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: helidon-tracing
  template:
    metadata:
      labels:
        app: helidon-tracing
        version: v1
    spec:
      containers:
        - name: helidon-tracing
          image: helidon-tracing-se
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080

• A service of type NodePort that serves the default routes on port 8080.
• A deployment with one replica of a pod.

kubectl apply -f ./tracing.yaml

kubectl get service/helidon-tracing

NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
helidon-tracing   NodePort   10.99.159.2   <none>        8080:31143/TCP   8s 

• A service of type NodePort that serves the default routes on port 31143.

curl http://localhost:31143/greet

{
  "message": "Hello World!"
}

Access the Jaeger UI at http://localhost:9412/jaeger and click on the refresh icon to see the trace that was just created.

You can now delete the Kubernetes resources just created during this example.

kubectl delete -f ./jaeger.yaml
kubectl delete -f ./tracing.yaml
kubectl delete service jaeger-external
docker rm -f jaeger

A listener cannot affect the lifecycle of a span or scope it is notified about, but it can add tags and events and update the baggage associated with a span. Often a listener does additional work that does not change the span or scope such as logging a message.

When Helidon invokes the listener’s methods it passes proxies for the Span.Builder, Span, and Scope arguments. These proxies limit the access the listener has to the span builder, span, or scope, as summarized in the following table. If a listener method tries to invoke a forbidden operation, the proxy throws a SpanListener.ForbiddenOperationException and Helidon then logs a WARNING message describing the invalid operation invocation.

The following tables list specifically what operations the proxies permit.

† Helidon returns the unwrapped object, not a proxy for it.

† Helidon returns the unwrapped object, not a proxy to it.

Explicitly Registering a Listener on a Tracer

Automatically Registering a Listener on all Tracer Instances