Helidon SE Tracing Guide

Helidon SE Tracing Guide: This guide describes how to create a sample Helidon SE project that can be used to run some basic examples using tracing with a Helidon SE application.

What You Need

For this 30 minute tutorial, you will need the following:

A Helidon {upper-case-flavor} Application	You can use your own application or use the Helidon {upper-case-flavor} Quickstart to create a sample application.
Java SE 11 (Open JDK 11)	Helidon requires Java 11+.
Maven 3.6.1+	Helidon requires Maven 3.6.1+.
Docker 18.09+	You need Docker if you want to build and deploy Docker containers.
Kubectl 1.16.5+	If you want to deploy to Kubernetes, you need `kubectl` and a Kubernetes cluster (you can install one on your desktop).

Verify Prerequisites

java -version
mvn --version
docker --version
kubectl version --short

Copied

Setting JAVA_HOME

# On Mac
export JAVA_HOME=`/usr/libexec/java_home -v 11`

# On Linux
# Use the appropriate path to your JDK
export JAVA_HOME=/usr/lib/jvm/jdk-11

Copied

Introduction

Distributed tracing is a critical feature of micro-service based applications, since it traces workflow both within a service and across multiple services. This provides insight to sequence and timing data for specific blocks of work, which helps you identify performance and operational issues. Helidon SE includes support for distributed tracing through the OpenTracing API. Tracing is integrated with WebServer, gRPC Server, and Security using either the Zipkin or Jaeger tracers.

Tracing Concepts

This section explains a few concepts that you need to understand before you get started with tracing. In the context of this document, a service is synonymous with an application. A span is the basic unit of work done within a single service, on a single host. Every span has a name, starting timestamp, and duration. For example, the work done by a REST endpoint is a span. A span is associated to a single service, but its descendants can belong to different services and hosts. A trace contains a collection of spans from one or more services, running on one or more hosts. For example, if you trace a service endpoint that calls another service, then the trace would contain spans from both services. Within a trace, spans are organized as a directed acyclic graph (DAG) and can belong to multiple services, running on multiple hosts. The OpenTracing Data Model describes the details at The OpenTracing Semantic Specification. Spans are automatically created by Helidon as needed during execution of the REST request.

Getting Started with Tracing

The examples in this guide demonstrate how to integrate tracing with Helidon, how to view traces, how to trace across multiple services, and how to integrate with tracing with Kubernetes. All examples use Zipkin and traces will be viewed using both the Zipkin API and UI.

Create a Sample Helidon SE Project

Use the Helidon SE Maven archetype to create a simple project that can be used for the examples in this guide.

Run the Maven archetype:

mvn -U archetype:generate -DinteractiveMode=false \
    -DarchetypeGroupId=io.helidon.archetypes \
    -DarchetypeArtifactId=helidon-quickstart-se \
    -DarchetypeVersion=2.6.14 \
    -DgroupId=io.helidon.examples \
    -DartifactId=helidon-quickstart-se \
    -Dpackage=io.helidon.examples.quickstart.se

Copied

The project will be built and run from the helidon-quickstart-se directory:

cd helidon-quickstart-se

Copied

Set up Zipkin

First, you need to run the Zipkin tracer. Helidon will communicate with this tracer at runtime.

Run Zipkin within a docker container, then check the Zipkin server health:

docker run -d --name zipkin -p 9411:9411 openzipkin/zipkin

Copied

Run the Zipkin docker image named openzipkin/zipkin.

Check the Zipkin server health:

curl http://localhost:9411/health 
...
{
  "status": "UP", 
  "zipkin": {
    "status": "UP",
    "details": {
      "InMemoryStorage{}": {
        "status": "UP"
      }
    }
  }
}

Copied

Invoke the Zipkin REST API to check the Zipkin server health.
All status fields should be UP.

Enable Tracing in the Helidon Application

Update the pom.xml file and add the following Zipkin dependency to the <dependencies> section (not <dependencyManagement>). This will enable Helidon to use Zipkin at the default host and port, localhost:9411.

Add the following dependency to pom.xml:

<dependency>
    <groupId>io.helidon.tracing</groupId>
    <artifactId>helidon-tracing</artifactId>
</dependency>
<dependency>
    <groupId>io.helidon.tracing</groupId>
    <artifactId>helidon-tracing-zipkin</artifactId>
</dependency>

Copied

All spans sent by Helidon to Zipkin need to be associated with a service. Specify the service name below.

Add the following line to resources/application.yaml:

tracing:
  service: helidon-se-1

Copied

Update the Main class; Add Tracer to the WebServer builder

import io.helidon.tracing.TracerBuilder; 
...
WebServer server = WebServer.builder(createRouting(config))
                .config(config.get("server"))
                .tracer(TracerBuilder.create(config.get("tracing")).build()) 
                .addMediaSupport(JsonpSupport.create())
                .build();

Copied

Add a new import statement.
Build and register a Tracer object using the tracing configuration.

Update the GreetService class; 1) Add a new import and 2) Replace the getDefaultMessageHandler method:

import io.opentracing.Span; 
...
    private void getDefaultMessageHandler(ServerRequest request,
                                   ServerResponse response) {

        var spanBuilder = request.tracer()  
                .buildSpan("getDefaultMessageHandler");  
        request.spanContext().ifPresent(spanBuilder::asChildOf);  
        Span span = spanBuilder.start();  

        try {
            sendResponse(response, "World");
        } finally {
            span.finish();  
        }
    }

Copied

Add new import statement.
Get the Tracer object from the request.
Build a new span named getDefaultMessageHandler.
Make the new span a child of the current span.
Start the span. The current timestamp is used as the starting time for the span.
Finish the span. The current timestamp is used as the ending time for the span.

Build the application, skipping unit tests, then run it:

mvn package -DskipTests=true
java -jar target/helidon-quickstart-se.jar

Copied

Run the curl command in a new terminal window and check the response:

curl http://localhost:8080/greet
...
{
  "message": "Hello World!"
}

Copied

Viewing Tracing Using Zipkin REST API

Because you had tracing enabled, the previous /greet endpoint invocation resulted in a new trace being created. Let’s get the trace data that was generated using the Zipkin API. First, get the service information.

Run the curl command and check the response:

curl http://localhost:9411/api/v2/services
...
["helidon-se-1"]

Copied

This is the tracing service name specified in resources/application.yaml.

Each span used by a service has a name, which is unique within a trace. If you invoke the /greet endpoint multiple times, you will still get the same set of names.

Invoke the endpoint below and check the response:


curl -X GET "http://localhost:9411/api/v2/spans?serviceName=helidon-se-1" -H "accept: application/json"
...
[ 
  "content-write",
  "getdefaultmessagehandler",
  "http request"
]

Copied

Get the span names for the helidon-se-1 service.
These are the span names. If you invoke the /greet endpoint again, then invoke the /spans endpoint, you will get the same response.

Next, get the spans in the trace as shown below.

Invoke the endpoint below and check the response:


curl -X GET "http://localhost:9411/api/v2/traces?serviceName=helidon-se-1&limit=1" -H "accept: application/json"
...
[
  [ 
    {
      "traceId": "f193adb3f2bab3b3",
      "parentId": "f193adb3f2bab3b3", 
      "id": "1536021daf3845e1",
      "kind": "SERVER",
      "name": "content-write",
      "timestamp": 1568245972222815,
      "duration": 527,
      "localEndpoint": {
        "serviceName": "helidon-se-1",
        "ipv4": "192.168.1.115"
      },
      "tags": {
        "response.type": "org.glassfish.json.JsonObjectBuilderImpl$JsonObjectImpl"
      }
    },
...
(truncated)
]

Copied

Get the newest trace only, using the limit=1 query param. There are other query params that let you restrict results to a specific time window.
The request will return 3 spans, one for each name.
Each span has a parentId field, except the http request span, which is the root.

Viewing Tracing Using Zipkin UI

The tracing output data is verbose and can be difficult to interpret using the REST API, especially since it represents a structure of spans. Zipkin provides a web-based UI at http://localhost:9411/zipkin, where you can see a visual representation of the same data and the relationship between spans within a trace.

Click on the UI refresh button (the search icon) as shown in the image below. Notice that you can change the look-back time to restrict the trace list.

Trace refresh

The image below shows the trace summary, including start time and duration of each trace. There are two traces, each one generated in response to a curl http://localhost:8080/greet invocation. The oldest trace will have a much longer duration since there is one-time initialization that occurs.

Tracing list view

Click on a trace and you will see the trace detail page where the spans are listed. You can clearly see the root span and the relationship among all the spans in the trace, along with timing information.

Trace detail page

A parent span might not depend on the result of the child. This is called a FollowsFrom reference, see Open Tracing Semantic Spec.

You can examine span details by clicking on the span row. Refer to the image below, which shows the span details, including timing information. You can see times for each space relative to the root span. These rows are annotated with Server Start and Server Finish, as shown in the third column.

Span detail page

Tracing Across Services

Helidon automatically traces across services, providing that the services use the same tracer, for example, the same instance of Zipkin. This means a single trace can include spans from multiple services and hosts. OpenTracing uses a SpanContext to propagate tracing information across process boundaries. When you make client API calls, Helidon will internally call OpenTracing APIs to propagate the SpanContext. There is nothing you need to do in your application to make this work.

To demonstrate distributed tracing, you will need to create a second project, where the server listens on port 8081. Create a new root directory to hold this new project, then do the following steps, similar to what you did at the start of this guide:

Create the Second Service

Run the Maven archetype:

mvn -U archetype:generate -DinteractiveMode=false \
    -DarchetypeGroupId=io.helidon.archetypes \
    -DarchetypeArtifactId=helidon-quickstart-se \
    -DarchetypeVersion=2.6.14 \
    -DgroupId=io.helidon.examples \
    -DartifactId=helidon-quickstart-se-2 \
    -Dpackage=io.helidon.examples.quickstart.se

Copied

The project will be built and run from the helidon-quickstart-se directory:

cd helidon-quickstart-se-2

Copied

Add the following dependency to pom.xml:

<dependency>
    <groupId>io.helidon.tracing</groupId>
    <artifactId>helidon-tracing</artifactId>
</dependency>
<dependency>
    <groupId>io.helidon.tracing</groupId>
    <artifactId>helidon-tracing-zipkin</artifactId>
</dependency>

Copied

Replace resources/application.yaml with the following:

app:
  greeting: "Hello From SE-2"

tracing:
  service: "helidon-se-2"

server:
  port: 8081
  host: 0.0.0.0

Copied

Update the Main class; Add Tracer to the WebServer builder

import io.helidon.tracing.TracerBuilder;
...
WebServer server = WebServer.builder(createRouting(config))
                .config(config.get("server"))
                .tracer(TracerBuilder.create(config.get("tracing")).build())
                .addMediaSupport(JsonpSupport.create())
                .build();

Copied

Update the GreetService class; 1) Add new import and 2) Replace the getDefaultMessageHandler method:

import io.opentracing.Span;
...
    private void getDefaultMessageHandler(ServerRequest request,
                                   ServerResponse response) {

        var spanBuilder = request.tracer()
                .buildSpan("getDefaultMessageHandler");
        request.spanContext().ifPresent(spanBuilder::asChildOf);
        Span span = spanBuilder.start();

        try {
            sendResponse(response, "World");
        } finally {
            span.finish();
        }
    }

Copied

Build the application, skipping unit tests, then run it:

mvn package -DskipTests=true
java -jar target/helidon-quickstart-se-2.jar

Copied

Run the curl command in a new terminal window and check the response (notice the port is 8081) :

curl http://localhost:8081/greet
...
{
  "message": "Hello From SE-2 World!"
}

Copied

Modify the First Service

Once you have validated that the second service is running correctly, you need to modify the original application to call it.

Add the following dependency to pom.xml:

<dependency>
    <groupId>io.helidon.security.integration</groupId>
    <artifactId>helidon-security-integration-jersey</artifactId>
</dependency>
<dependency>
    <groupId>io.helidon.tracing</groupId>
    <artifactId>helidon-tracing-jersey-client</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.core</groupId>
    <artifactId>jersey-client</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.inject</groupId>
    <artifactId>jersey-hk2</artifactId>
</dependency>

Copied

Replace the GreetService class with the following code:

package io.helidon.examples.quickstart.se;

import io.helidon.common.http.Http;
import io.helidon.config.Config;
import io.helidon.tracing.jersey.client.ClientTracingFilter;
import io.helidon.webserver.Routing;
import io.helidon.webserver.ServerRequest;
import io.helidon.webserver.ServerResponse;
import io.helidon.webserver.Service;
import io.opentracing.Span;
import java.util.Collections;
import java.util.concurrent.atomic.AtomicReference;
import javax.json.Json;
import javax.json.JsonBuilderFactory;
import javax.json.JsonObject;
import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.client.Invocation;
import javax.ws.rs.client.WebTarget;

public class GreetService implements Service {

  private final AtomicReference<String> greeting = new AtomicReference<>();
  private WebTarget webTarget;
  private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Collections.emptyMap());

  GreetService(Config config) {
    greeting.set(config.get("app.greeting").asString().orElse("Ciao"));

    Client jaxRsClient = ClientBuilder.newBuilder().build();

    webTarget = jaxRsClient.target("http://localhost:8081/greet");
  }

  @Override
  public void update(Routing.Rules rules) {
    rules
        .get("/", this::getDefaultMessageHandler)
        .get("/outbound", this::outboundMessageHandler) 
        .put("/greeting", this::updateGreetingHandler);
  }

  private void getDefaultMessageHandler(ServerRequest request, ServerResponse response) {

    var spanBuilder = request.tracer()
                .buildSpan("getDefaultMessageHandler");
    request.spanContext().ifPresent(spanBuilder::asChildOf);
    Span span = spanBuilder.start();

    try {
      sendResponse(response, "World");
    } finally {
      span.finish();
    }
  }

  private void sendResponse(ServerResponse response, String name) {
    String msg = String.format("%s %s!", greeting.get(), name);

    JsonObject returnObject = JSON.createObjectBuilder().add("message", msg).build();
    response.send(returnObject);
  }

  private void updateGreetingFromJson(JsonObject jo, ServerResponse response) {

    if (!jo.containsKey("greeting")) {
      JsonObject jsonErrorObject =
          JSON.createObjectBuilder().add("error", "No greeting provided").build();
      response.status(Http.Status.BAD_REQUEST_400).send(jsonErrorObject);
      return;
    }

    greeting.set(jo.getString("greeting"));
    response.status(Http.Status.NO_CONTENT_204).send();
  }

  private void outboundMessageHandler(ServerRequest request, ServerResponse response) {
    Invocation.Builder requestBuilder = webTarget.request();

    
    var spanBuilder = request.tracer()
                .buildSpan("outboundMessageHandler");
    request.spanContext().ifPresent(spanBuilder::asChildOf);
    Span span = spanBuilder.start();

    try {
      requestBuilder.property(
          ClientTracingFilter.CURRENT_SPAN_CONTEXT_PROPERTY_NAME, request.spanContext());  

      requestBuilder   
          .rx()
          .get(String.class)
          .thenAccept(response::send)
          .exceptionally(
              throwable -> {
                // process exception
                response.status(Http.Status.INTERNAL_SERVER_ERROR_500);
                response.send("Failed with: " + throwable);
                return null;
              });
    } finally {
      span.finish();   
    }
  }

  private void updateGreetingHandler(ServerRequest request, ServerResponse response) {
    request.content().as(JsonObject.class).thenAccept(jo -> updateGreetingFromJson(jo, response));
  }
}

Copied

Add outboundMessageHandler to the routing rules.
Create and start a span that is a child of the current span.
Set a property with the SpanContext.
Invoke the second service.
Stop the span.

Build and run the application, then invoke the endpoint and check the response:

curl -i http://localhost:8080/greet/outbound 
...
{
  "message": "Hello From SE-2 World!" 
}

Copied

The request went to the service on 8080, which then invoked the service at 8081 to get the greeting.
Notice the greeting came from the second service.

Refresh the Zipkin UI trace listing page and notice that there is a trace across two services.

Tracing multiple service list view

Click on the trace with two services to see the detail view.

Tracing across multiple services detail view

In the image above, you can see that the trace includes spans from two services. You will notice there is a gap before the sixth span, which is a get operation. This is a one-time client initialization delay. Run the /outbound curl command again and look at the new trace to see that the delay no longer exists.

You can now stop your second service, it is not longer used in this guide.

Integration with Kubernetes

The following example demonstrate how to use Zipkin from a Helidon application running in Kubernetes.

Replace the tracing configuration in resources/application.yaml with the following:


tracing:
  service: helidon-se-1
  host: zipkin

Copied

Helidon service helidon-se-1 will connect to the Zipkin server at host name zipkin.

Stop the application and build the docker image for your application:

docker build -t helidon-tracing-se .

Copied

Deploy Zipkin into Kubernetes

Create the Kubernetes YAML specification, named zipkin.yaml, with the following contents:

apiVersion: v1
kind: Service
metadata:
  name: zipkin
spec:
  ports:
    - port: 9411
      protocol: TCP
  selector:
    app: zipkin
---
kind: Pod
apiVersion: v1
metadata:
  name: zipkin
  labels:
    app: zipkin
spec:
  containers:
    - name: zipkin
      image: openzipkin/zipkin
      imagePullPolicy: IfNotPresent
      ports:
        - containerPort: 9411

Copied

Create the Zipkin pod and ClusterIP service:

kubectl apply -f ./zipkin.yaml

Copied

Create a Zipkin external server to view the UI and expose it on port 9142:

kubectl expose pod  zipkin --name=zipkin-external --port=9412 --target-port=9411 --type=LoadBalancer

Copied

Navigate to http://localhost:9412/zipkin to validate that you can access Zipkin running in Kubernetes. It may take a few seconds before it is ready.

Deploy Your Helidon Application into Kubernetes

Create the Kubernetes YAML specification, named tracing.yaml, with the following contents:

kind: Service
apiVersion: v1
metadata:
  name: helidon-tracing 
  labels:
    app: helidon-tracing
spec:
  type: NodePort
  selector:
    app: helidon-tracing
  ports:
    - port: 8080
      targetPort: 8080
      name: http
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: helidon-tracing
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: helidon-tracing
  template:
    metadata:
      labels:
        app: helidon-tracing
        version: v1
    spec:
      containers:
        - name: helidon-tracing
          image: helidon-tracing-se
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080

Copied

A service of type NodePort that serves the default routes on port 8080.
A deployment with one replica of a pod.

Create and deploy the application into Kubernetes:

kubectl apply -f ./tracing.yaml

Copied

Access Your Application and the Zipkin Trace

Get the application service information:

kubectl get service/helidon-tracing

Copied

NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
helidon-tracing   NodePort   10.99.159.2   <none>        8080:31143/TCP   8s

Copied

A service of type NodePort that serves the default routes on port 31143.

Verify the tracing endpoint using port 31143, your port will likely be different:

curl http://localhost:31143/greet
...
{
  "message": "Hello World!"
}

Copied

Access the Zipkin UI at http://localhost:9412/zipkin and click on the refresh icon to see the trace that was just created.

Cleanup

You can now delete the Kubernetes resources that were just created during this example.

Delete the Kubernetes resources:

kubectl delete -f ./zipkin.yaml
kubectl delete -f ./tracing.yaml
kubectl delete service zipkin-external
docker rm -f zipkin

Copied

Summary

This guide has demonstrated how to use the Helidon SE tracing feature with Zipkin. You have learned to do the following:

Enable tracing within a service
Use tracing with JAX-RS
Use the Zipkin REST API and UI
Use tracing across multiple services
Integrate tracing with Kubernetes

Refer to the following references for additional information:

MicroProfile OpenTracing specification at https://github.com/eclipse/microprofile-opentracing/releases/tag/1.3
Helidon Javadoc at https://helidon.io/docs/latest/apidocs/index.html?overview-summary.html