Tracing

Overview

Distributed tracing is a critical feature of micro-service based applications, since it traces workflow both within a service and across multiple services. This provides insight to sequence and timing data for specific blocks of work, which helps you identify performance and operational issues. Helidon includes support for distributed tracing through its own API, backed by either through the OpenTelemetry API, or by OpenTracing API. Tracing is integrated with WebServer, gRPC Server, and Security.

Maven Coordinates

To enable Helidon Tracing add the following dependency to your project’s pom.xml (see Managing Dependencies).

<dependency>
    <groupId>io.helidon.tracing</groupId>
    <artifactId>helidon-tracing</artifactId>
</dependency>

Copied

Usage

This section explains a few concepts that you need to understand before you get started with tracing.

In the context of this document, a service is synonymous with an application.
A span is the basic unit of work done within a single service, on a single host. Every span has a name, starting timestamp, and duration. For example, the work done by a REST endpoint is a span. A span is associated to a single service, but its descendants can belong to different services and hosts.
A trace contains a collection of spans from one or more services, running on one or more hosts. For example, if you trace a service endpoint that calls another service, then the trace would contain spans from both services. Within a trace, spans are organized as a directed acyclic graph (DAG) and can belong to multiple services, running on multiple hosts.

Support for specific tracers is abstracted. Your application can depend on the Helidon abstraction layer and provide a specific tracer implementation as a Java ServiceLoader service. Helidon provides such an implementation for:

OpenTracing tracers, either using the GlobalTracer, provider resolver approach, or explicitly using Zipkin tracer
OpenTelemetry tracers, either using the global OpenTelemetry instance, or explicitly using Jaeger tracer

Setup WebServer

Configuring OpenTracing Tracer

ServerConfiguration.builder()
                   .tracer(TracerBuilder.create("my-application")                    
                                 .collectorUri(URI.create("http://10.0.0.18:9411"))  
                                 .build())
                   .build()

Copied

The name of the application (service) to associate with the tracing events
The endpoint for tracing events, specific to the tracer used, usually loaded from Config

Setup gRPC Server

Configuring Tracer

Tracer tracer = TracerBuilder.create("Server")
        .collectorUri(URI.create("http://10.0.0.18:9411"))   
        .build();

Copied

If using zipkin tracing system, the endpoint would be: http://10.0.0.18:9411/api/v2/spans

Configuring Tracing Attributes

GrpcTracingConfig tracingConfig = new GrpcTracingConfig.Builder()
                .withStreaming()
                .withVerbosity()
                .withTracedAttributes(ServerRequestAttribute.CALL_ATTRIBUTES,
                     ServerRequestAttribute.HEADERS,
                     ServerRequestAttribute.METHOD_NAME)
                .build();

Copied

Configuring gRPC Server

GrpcServerConfiguration serverConfig = GrpcServerConfiguration.builder().port(0)
                .tracer(tracer)
                .tracingConfig(tracingConfig)
                .build();

Copied

Creating Custom Spans

To create a custom span from tracer:

Span span = tracer.spanBuilder("name")  
        .tag("key", "value")
        .start();
try (...){                              
    //do some work
    span.end();
} catch (Exception e) {                 
    span.end(e);
}

Copied

Create span from tracer.
Do some work and end span.
End span with exception.

Helidon Spans

Traced spans

The following table lists all spans traced by Helidon components:

component	span name	description
`web-server`	`HTTP Request`	The overall span of the Web Server from request initiation until response Note that in `Zipkin` the name is replaced with `jax-rs` span name if `jax-rs` tracing is used.
`web-server`	`content-read`	Span for reading the request entity
`web-server`	`content-write`	Span for writing the response entity
`security`	`security`	Processing of request security
`security`	`security:atn`	Span for request authentication
`security`	`security:atz`	Span for request authorization
`security`	`security:response`	Processing of response security
`security`	`security:outbound`	Processing of outbound security
`jax-rs`	A generated name	Span for the resource method invocation, name is generated from class and method name
`jax-rs`	`jersey-client-call`	Span for outbound client call

Some of these spans log to the span. These log events can be (in most cases) configured:

span name	log name	configurable	enabled by default	description
`HTTP Request`	`handler.class`	YES	YES	Each handler has its class and event logged
`security`	`status`	YES	YES	Logs either "status: PROCEED" or "status: DENY"
`security:atn`	`security.user`	YES	NO	The username of the user if logged in
`security:atn`	`security.service`	YES	NO	The name of the service if logged in
`security:atn`	`status`	YES	YES	Logs the status of security response (such as `SUCCESS`)
`security:atz`	`status`	YES	YES	Logs the status of security response (such as `SUCCESS`)
`security:outbound`	`status`	YES	YES	Logs the status of security response (such as `SUCCESS`)

There are also tags that are set by Helidon components. These are not configurable.

span name	tag name	description
`HTTP Request`	`component`	name of the component - `helidon-webserver`, or `jaxrs` when using MP
`HTTP Request`	`http.method`	HTTP method of the request, such as `GET`, `POST`
`HTTP Request`	`http.status_code`	HTTP status code of the response
`HTTP Request`	`http.url`	The path of the request (for SE without protocol, host and port)
`HTTP Request`	`error`	If the request ends in error, this tag is set to `true`, usually accompanied by logs with details
`content-read`	`requested.type`	Type (class) of the requested entity (if entity is read)
`content-write`	`response.type`	Type (class) of the entity being sent (if entity is sent)
`security`	`security.id`	ID of the security context created for this request (if security is used)
`jersey-client-call`	`http.method`	HTTP method of the client request
`jersey-client-call`	`http.status_code`	HTTP status code of client response
`jersey-client-call`	`http.url`	Full URL of the request (such as `http://localhost:8080/greet`)

Configuration

The following configuration should be supported by all tracer implementations (if feasible)

Jaeger tracer configuration.

Type: io.helidon.tracing.Tracer

This is a standalone configuration type, prefix from configuration root: tracing

Configuration options

Optional configuration options

key	type	default value	description
`boolean-tags`	Map<string, boolean>		Tracer level tags that get added to all reported spans.
`client-cert-pem`	Resource		Certificate of client in PEM format.
`enabled`	boolean	`true`	When enabled, tracing will be sent. If enabled is false, tracing should use a no-op tracer.
`exporter-timeout-millis`	Duration	`10000`	Timeout of exporter requests.
`global`	boolean	`true`	When enabled, the created instance is also registered as a global tracer.
`host`	string		Host to use to connect to tracing collector. Default is defined by each tracing integration.
`int-tags`	Map<string, int>		Tracer level tags that get added to all reported spans.
`path`	string		Path on the collector host to use when sending data to tracing collector. Default is defined by each tracing integration.
`port`	int		Port to use to connect to tracing collector. Default is defined by each tracing integration.
`private-key-pem`	Resource		Private key in PEM format.
`protocol`	string		Protocol to use (such as `http` or `https`) to connect to tracing collector. Default is defined by each tracing integration.
`sampler-param`	Number	`1`	The sampler parameter (number).
`sampler-type`	SamplerType (CONSTANT, RATIO)	`CONSTANT`	Sampler type. See <a href="https://www.jaegertracing.io/docs/latest/sampling/#client-sampling-configuration">Sampler types</a>.
`service`	string		Service name of the traced service.
`tags`	Map<string, string>		Tracer level tags that get added to all reported spans.
`trusted-cert-pem`	Resource		Trusted certificates in PEM format.

Traced Spans Configuration

Each component and its spans can be configured using Config. The traced configuration has the following layers:

TracingConfig - the overall configuration of traced components of Helidon
ComponentTracingConfig - a component of Helidon that traces spans (such as web-server, security, jax-rs)
SpanTracingConfig - a single traced span within a component (such as security:atn)
SpanLogTracingConfig - a single log event on a span (such as security.user in span security:atn)

The components using tracing configuration use the TracingConfigUtil. This uses the io.helidon.common.Context to retrieve current configuration.

Configuration using builder

Builder approach, example that disables a single span log event:

Configure tracing using a builder

TracingConfig.builder()
     .addComponent(ComponentTracingConfig.builder("web-server")
             .addSpan(SpanTracingConfig.builder("HTTP Request")
                     .addSpanLog(SpanLogTracingConfig.builder("content-write").enabled(false).build())
                     .build())
             .build())
     .build()

Copied

Configuration using Helidon Config

Tracing configuration can be defined in a config file.

Tracing configuration

tracing:
    components:
      web-server:
        spans:
          - name: "HTTP Request"
            logs:
              - name: "content-write"
                enabled: false

Copied

Use the configuration in web server

routing.register(WebTracingConfig.create(config.get("tracing")));

Copied

Path-based configuration in Helidon Web Server

For Web Server we have path-based support for configuring tracing, in addition to the configuration described above.

Configuration of path can use any path string supported by the Web Server. The configuration itself has the same possibilities as traced configuration described above. The path specific configuration will be merged with global configuration (path is the "newer" configuration, global is the "older")

Configuration in YAML

tracing:
  paths:
    - path: "/favicon.ico"
      enabled: false
    - path: "/metrics"
      enabled: false
    - path: "/health"
      enabled: false
    - path: "/greet"
      components:
        web-server:
          spans:
          - name: "content-read"
            new-name: "read"
            enabled: false

Copied

Configuration with Web Server

routingBuilder.register(WebTracingConfig.create(config.get("tracing"));

Copied

Configuration with Web Server using a builder

routingBuilder.register(WebTracingConfig.builder()
    .addPathConfig(PathTracingConfig.builder()
        .path("/metrics")
        .tracingConfig(TracingConfig.DISABLED)
        .build();
    .build());

Copied

Renaming top level span using request properties

To have a nicer overview in search pane of a tracer, you can customize the top-level span name using configuration.

Example:

Configuration in YAML

tracing:
  components:
    web-server:
      spans:
      - name: "HTTP Request"
        new-name: "HTTP %1$s %2$s"

Copied

This is supported ONLY for the span named "HTTP Request" on component "web-server".

Parameters provided:

Method - HTTP method
Path - path of the request (such as '/greet')
Query - query of the request (may be null)

Additional Information

Span Propagation

Span propagation is supported with Helidon WebClient (and with Jersey client, though it is blocking and not suitable for reactive implementations). Tracing propagation is automatic as long as the current span context is available in Helidon Context (which is automatic when running within a WebServer request).

<dependency>
    <groupId>io.helidon.webclient</groupId>
    <artifactId>helidon-webclient</artifactId>
</dependency>
<dependency>
    <groupId>io.helidon.webclient</groupId>
    <artifactId>helidon-webclient-tracing</artifactId>
</dependency>

Copied

Tracing propagation with Helidon WebClient

WebClient client = WebClient.builder()
    .addService(WebClientTracing.create())
    .build();

Single<String> response = client.get()
    .uri(uri)
    .request(String.class);

Copied

Zipkin Tracing

<dependency>
    <groupId>io.helidon.tracing</groupId>
    <artifactId>helidon-tracing-zipkin</artifactId>
</dependency>

Copied

Configuring Zipkin

Zipkin tracer configuration

Type: io.opentracing.Tracer

This is a standalone configuration type, prefix from configuration root: tracing

Configuration options

Optional configuration options

key	type	default value	description
`api-version`	Version (V1, V2)	`V2`	Version of Zipkin API to use. Defaults to Version#V2.
`boolean-tags`	Map<string, boolean>		Tracer level tags that get added to all reported spans.
`enabled`	boolean	`true`	When enabled, tracing will be sent. If enabled is false, tracing should use a no-op tracer.
`global`	boolean	`true`	When enabled, the created instance is also registered as a global tracer.
`host`	string		Host to use to connect to tracing collector. Default is defined by each tracing integration.
`int-tags`	Map<string, int>		Tracer level tags that get added to all reported spans.
`path`	string		Path on the collector host to use when sending data to tracing collector. Default is defined by each tracing integration.
`port`	int		Port to use to connect to tracing collector. Default is defined by each tracing integration.
`protocol`	string		Protocol to use (such as `http` or `https`) to connect to tracing collector. Default is defined by each tracing integration.
`service`	string		Service name of the traced service.
`tags`	Map<string, string>		Tracer level tags that get added to all reported spans.

The following is an example of a Zipkin configuration, specified in the YAML format.

tracing:
  service: "helidon-service"
  protocol: "https"
  host: "zipkin"
  port: 9987
  api-version: 1
  # this is the default path for API version 2
  path: "/api/v2/spans"
  tags:
    tag1: "tag1-value"
    tag2: "tag2-value"
  boolean-tags:
    tag3: true
    tag4: false
  int-tags:
    tag5: 145
    tag6: 741

Copied

Example of Zipkin trace:

Jaeger Tracing

<dependency>
    <groupId>io.helidon.tracing</groupId>
    <artifactId>helidon-tracing-jaeger</artifactId>
</dependency>

Copied

Configuring Jaeger

Jaeger changed its client implementation, so some Jaeger settings exposed by earlier releases of Helidon are no longer available. Please note the currently-supported settings in the table below.

Jaeger tracer configuration.

Type: io.helidon.tracing.Tracer

This is a standalone configuration type, prefix from configuration root: tracing

Configuration options

Optional configuration options

key	type	default value	description
`boolean-tags`	Map<string, boolean>		Tracer level tags that get added to all reported spans.
`client-cert-pem`	Resource		Certificate of client in PEM format.
`enabled`	boolean	`true`	When enabled, tracing will be sent. If enabled is false, tracing should use a no-op tracer.
`exporter-timeout-millis`	Duration	`10000`	Timeout of exporter requests.
`global`	boolean	`true`	When enabled, the created instance is also registered as a global tracer.
`host`	string		Host to use to connect to tracing collector. Default is defined by each tracing integration.
`int-tags`	Map<string, int>		Tracer level tags that get added to all reported spans.
`path`	string		Path on the collector host to use when sending data to tracing collector. Default is defined by each tracing integration.
`port`	int		Port to use to connect to tracing collector. Default is defined by each tracing integration.
`private-key-pem`	Resource		Private key in PEM format.
`propagation`	`JAEGER`	addPropagation	Propagation type (`jaeger`, `b3`, `b3_single`, `w3c`). Jaeger is the default, `b3` is for compatibility with Zipkin (using multiple headers), `b3_single` using a single header.
`protocol`	string		Protocol to use (such as `http` or `https`) to connect to tracing collector. Default is defined by each tracing integration.
`sampler-param`	Number	`1`	The sampler parameter (number).
`sampler-type`	SamplerType (CONSTANT, RATIO)	`CONSTANT`	Sampler type. See <a href="https://www.jaegertracing.io/docs/latest/sampling/#client-sampling-configuration">Sampler types</a>.
`service`	string		Service name of the traced service.
`tags`	Map<string, string>		Tracer level tags that get added to all reported spans.
`trusted-cert-pem`	Resource		Trusted certificates in PEM format.

The following is an example of a Jaeger configuration, specified in the YAML format.

The the Jaeger OpenTelemetry client uses port 14250, but you can override this value if needed. The default is defined by each tracing integration.

tracing:
    service: "helidon-full-http"
    protocol: "https"
    host: "jaeger"
    port: 14250

Copied

Jaeger Tracing Metrics

As the Jaeger Tracing section describes, you can use Jaeger tracing in your Helidon application.

Contents

Overview

Maven Coordinates

Usage

Setup WebServer

Setup gRPC Server

Creating Custom Spans

Helidon Spans

Traced spans

Configuration

Configuration options

Traced Spans Configuration

Configuration using builder

Configuration using Helidon Config

Path-based configuration in Helidon Web Server

Renaming top level span using request properties

Additional Information

Span Propagation

Zipkin Tracing

Configuring Zipkin

Configuration options

Jaeger Tracing

Configuring Jaeger

Configuration options

Jaeger Tracing Metrics

Reference