• Overview

• Maven Coordinates

• Usage

• Configuration

• Examples

• Additional Information

  • Jaeger Tracing

  • Zipkin Tracing

  • OpenTelemetry Tracing

  • Responding to Span Life Cycle Events

• Reference

Distributed tracing is a critical feature of micro-service based applications, since it traces workflow both within a service and across multiple services. This provides insight to sequence and timing data for specific blocks of work, which helps you identify performance and operational issues. Helidon MP includes support for distributed tracing through the OpenTracing API. Tracing is integrated with WebServer and Security.

To enable MicroProfile Tracing, either add a dependency on the helidon-microprofile bundle or add the following dependency to your project’s pom.xml (see Managing Dependencies).

<dependency>
    <groupId>io.helidon.microprofile.tracing</groupId>
    <artifactId>helidon-microprofile-tracing</artifactId>
</dependency>

This section explains a few concepts that you need to understand before you get started with tracing.

• In the context of this document, a service is synonymous with an application.

• A span is the basic unit of work done within a single service, on a single host. Every span has a name, starting timestamp, and duration. For example, the work done by a REST endpoint is a span. A span is associated to a single service, but its descendants can belong to different services and hosts.

• A trace contains a collection of spans from one or more services, running on one or more hosts. For example, if you trace a service endpoint that calls another service, then the trace would contain spans from both services. Within a trace, spans are organized as a directed acyclic graph (DAG) and can belong to multiple services, running on multiple hosts. The OpenTracing Data Model describes the details at The OpenTracing Semantic Specification. Spans are automatically created by Helidon as needed during execution of the REST request. Additional spans can be added through MP annotation @Traced or through OpenTracing APIs.

The following table lists all spans traced by Helidon components:

Some of these spans log to the span. These log events can be (in most cases) configured:

There are also tags that are set by Helidon components. These are not configurable.

Enabling and Disabling Tracing

To disable Helidon tracing for web server and security:

tracing.components.web-server.enabled=false
tracing.components.security.enabled=false

To disables MP Tracing as by specification:

mp.opentracing.server.skip-pattern=.*

Tracing configuration can be defined in application.yaml file.

tracing:
  paths:
    - path: "/favicon.ico"
      enabled: false
    - path: "/metrics"
      enabled: false
    - path: "/health"
      enabled: false
  components:
    web-server:
      spans:
        - name: "HTTP Request"
          logs:
            - name: "content-write"
              enabled: false

Controlling Tracing Output

tracing.components.web-server.spans.0.name="HTTP Request"
tracing.components.web-server.spans.0.new-name: "HTTP %1$s %2$s"

The examples in this guide demonstrate how to integrate tracing with Helidon, how to view traces, how to trace across multiple services, and how to integrate tracing with Kubernetes. All examples use Jaeger and traces will be viewed using both the Jaeger UI.

Set up Jaeger

docker run -d --name jaeger \                  
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.50

http://localhost:16686/search

Trace Across Services

Helidon automatically traces across services as long as the services use the same tracer, for example, the same instance of Jaeger. This means a single trace can include spans from multiple services and hosts. OpenTracing uses a SpanContext to propagate tracing information across process boundaries. When you make client API calls, Helidon will internally call OpenTracing APIs to propagate the SpanContext. There is nothing you need to do in your application to make this work.

To demonstrate distributed tracing, you will need to create a second project, where the server listens on port 8081. Create a new root directory to hold this new project, then do the following steps, similar to what you did at the start of this guide:

Create a second service

Run the Maven archetype:

mvn -U archetype:generate -DinteractiveMode=false \
    -DarchetypeGroupId=io.helidon.archetypes \
    -DarchetypeArtifactId=helidon-quickstart-mp \
    -DarchetypeVersion=4.4.1 \
    -DgroupId=io.helidon.examples \
    -DartifactId=helidon-quickstart-mp-2 \
    -Dpackage=io.helidon.examples.quickstart.mp

content_copy Copied

The project will be built and run from the helidon-quickstart-mp directory:

cd helidon-quickstart-mp-2

content_copy Copied

Add the following dependency to pom.xml:

<dependency>
    <groupId>io.helidon.tracing.providers</groupId>
    <artifactId>helidon-tracing-providers-jaeger</artifactId>
</dependency>

content_copy Copied

Replace META-INF/microprofile-config.properties with the following:

app.greeting=Hello From MP-2
tracing.service=helidon-mp-2

# Microprofile server properties
server.port=8081

content_copy Copied

Build the application, skipping unit tests, then run it:

mvn package -DskipTests=true
java -jar target/helidon-quickstart-mp-2.jar

content_copy Copied

Run the curl command in a new terminal window and check the response (notice the port is 8081) :

curl http://localhost:8081/greet

content_copy Copied

Response body

{
  "message": "Hello From MP-2 World!"
}

content_copy Copied

Modify the first service

Once you have validated that the second service is running correctly, you need to modify the original application to call it.

Replace the GreetResource class with the following code:

@Path("/greet")
@RequestScoped
public class GreetResource {

    @Uri("http://localhost:8081/greet")
    private WebTarget target; 

    private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Map.of());
    private final GreetingProvider greetingProvider;

    @Inject
    public GreetResource(GreetingProvider greetingConfig) {
        this.greetingProvider = greetingConfig;
    }

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public JsonObject getDefaultMessage() {
        return createResponse("World");
    }

    @GET
    @Path("/outbound") 
    public JsonObject outbound() {
        return target.request().accept(MediaType.APPLICATION_JSON_TYPE).get(JsonObject.class);
    }

    private JsonObject createResponse(String who) {
        String msg = String.format("%s %s!", greetingProvider.getMessage(), who);

        return JSON.createObjectBuilder().add("message", msg).build();
    }
}

content_copy Copied

• This is the WebTarget needed to send a request to the second service at port 8081.
• This is the new endpoint that will call the second service.

Build and run the application, then invoke the endpoint and check the response:

curl -i http://localhost:8080/greet/outbound 

content_copy Copied

• The request went to the service on 8080, which then invoked the service at 8081 to get the greeting.

Response body

{
  "message": "Hello From MP-2 World!" 
}

content_copy Copied

• Notice the greeting came from the second service.

Refresh the Jaeger UI trace listing page and notice that there is a trace across two services.

Tracing across multiple services detail view

In the image above, you can see that the trace includes spans from two services. You will notice there is a gap before the sixth span, which is a get operation. This is a one-time client initialization delay. Run the /outbound curl command again and look at the new trace to see that the delay no longer exists.

You can now stop your second service, it is no longer used in this guide.

The following example demonstrates how to use Jaeger from a Helidon application running in Kubernetes.

tracing:
  host: "jaeger"

docker build -t helidon-tracing-mp .

Deploy Jaeger into Kubernetes

apiVersion: v1
kind: Service
metadata:
  name: jaeger
spec:
  ports:
    - port: 16686
      protocol: TCP
  selector:
    app: jaeger
---
kind: Pod
apiVersion: v1
metadata:
  name: jaeger
  labels:
    app: jaeger
spec:
  containers:
    - name: jaeger
      image: jaegertracing/all-in-one
      imagePullPolicy: IfNotPresent
      ports:
        - containerPort: 16686

kubectl apply -f ./jaeger.yaml

kubectl expose pod jaeger --name=jaeger-external --port=16687 --target-port=16686 --type=LoadBalancer 

Deploy Your Helidon Application into Kubernetes

kind: Service
apiVersion: v1
metadata:
  name: helidon-tracing 
  labels:
    app: helidon-tracing
spec:
  type: NodePort
  selector:
    app: helidon-tracing
  ports:
    - port: 8080
      targetPort: 8080
      name: http
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: helidon-tracing
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: helidon-tracing
  template:
    metadata:
      labels:
        app: helidon-tracing
        version: v1
    spec:
      containers:
        - name: helidon-tracing
          image: helidon-tracing-mp
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080

kubectl apply -f ./tracing.yaml

Access Your Application and the Jaeger Trace

kubectl get service/helidon-tracing

NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
helidon-tracing   NodePort   10.99.159.2   <none>        8080:31143/TCP   8s 

curl http://localhost:31143/greet

{
  "message": "Hello World!"
}

Cleanup

kubectl delete -f ./jaeger.yaml
kubectl delete -f ./tracing.yaml
kubectl delete service jaeger-external
docker rm -f jaeger

Helidon MP fully supports MicroProfile OpenTracing. You can add custom spans using @Traced annotation on methods of CDI beans.

Note for invoking methods on same class: If you invoke a method on the same class, @Traced annotation would be ignored, as it is not invoked through a CDI proxy and as such cannot be intercepted. To make sure @Traced is honored, use it on JAX-RS resource methods and on CDI bean methods used from other beans.

Automated trace propagation is supported currently only with Jersey client.

Tracing propagation works automatically as long as you run within the scope of Helidon MP and use Helidon components to invoke external services.

Manual handling of traces in Jersey Client

Response response = client.target(serviceEndpoint)
        .request()
        // tracer should be provided unless available as GlobalTracer
        .property(TRACER_PROPERTY_NAME, tracer)
        .property(CURRENT_SPAN_CONTEXT_PROPERTY_NAME, spanContext)
        .get();

Jaeger Tracing

<dependency>
    <groupId>io.helidon.tracing</groupId>
    <artifactId>helidon-tracing-providers-jaeger</artifactId>
</dependency>

The following is an example of a Jaeger configuration, specified in the YAML format.

tracing:
    service: "helidon-full-http"
    protocol: "https"
    host: "jaeger"
    port: 14240

Jaeger Tracing Metrics

Zipkin Tracing

<dependency>
    <groupId>io.helidon.tracing.providers</groupId>
    <artifactId>helidon-tracing-providers-zipkin</artifactId>
</dependency>

The following is an example of a Zipkin configuration, specified in the YAML format.

tracing:
  zipkin:
    service: "helidon-service"
    protocol: "https"
    host: "zipkin"
    port: 9987
    api-version: 1
    # this is the default path for API version 2
    path: "/api/v2/spans"
    tags:
      tag1: "tag1-value"
      tag2: "tag2-value"
    boolean-tags:
      tag3: true
      tag4: false
    int-tags:
      tag5: 145
      tag6: 741

Example of Zipkin trace:

OpenTelemetry Tracing

<dependency>
    <groupId>io.helidon.tracing.providers</groupId>
    <artifactId>helidon-tracing-providers-opentelemetry</artifactId>
</dependency>

tracing.service=helidon-otel-tracing-example 
tracing.global=false                         
tracing.int-tags.0.example=1                 
tracing.tags.0.direction=north               

• Specifies the OpenTelemetry service name.
• Indicates the configured tracer should not be made the global tracer (defaults to true).
• Assigns an integer-valued tag example the value 1.
• Assigns a string-valued tag direction the value north.

By default, Helidon tracing support for OpenTelemetry uses OpenTelemetry’s OTLP gRPC exporter. Alternatively, you can choose to use OpenTelemetry’s HTTP exporter using protobuf by setting exporter-type to http/proto. To use other exporters OpenTelemetry offers, use the Helidon telemetry configuration instead of tracing.

Responding to Span Lifecycle Events

• MicroProfile Opentracing Specification

• Opentracing Project