Overview

Helidon SE support for OpenTelemetry configuration and semantic conventions as described below is currently an incubating feature. We intend to support it going forward, but we might withdraw the feature entirely or change its external API and behavior in backward-incompatible ways across dot releases.

Helidon SE supports OpenTelemetry in several important ways:

  • Implements the neutral Helidon tracing API using OpenTelemetry

  • Allows users to assign OpenTelemetry settings as follows:

    • Declaratively, using Helidon config under the top-level telemetry config key

    • Programmatically, using the OpenTelemetry SDK API and the Helidon OpenTelemetry API

  • Conforms to the OpenTelemetry semantic conventions for automatically-created spans

OpenTelemetry models observability as a set of signals. Each signal—​for example metrics, tracing, and logging—​is an origin of monitoring data, and each has configurable settings which control its behavior.

Helidon’s config support for OpenTelemetry has certain config attributes which apply to OpenTelemetry as a whole, others which pertain to individual signals, and still more which describe lower-level elements within a signal.

The Helidon OpenTelemetry configuration format, the Helidon OpenTelemetry API, and this documentation all follow this hierarchy:

This document describes how to configure each level in the hierarchy and covers general topics related to Helidon’s support of OpenTelemetry.

API

There are two APIs that might be useful to developers working with OpenTelemetry:

  • The Helidon OpenTelemetry API - useful for mapping configuration sources to Helidon builders and, ultimately, OpenTelemetry objects.

  • The OpenTelemetry API - useful for creating OpenTelemetry objects apart from Helidon configuration sources.

The types of the Helidon OpenTelemetry API correspond closely to the configuration structures described in later sections of this document. Application code can use Helidon OpenTelemetry builders to prepare and construct each of the configurable entities to ultimately prepare an OpenTelemetry instance set up according to the application’s needs.

That said, application code can equally well use the OpenTelemetry API and its builders to prepare the OpenTelemetry instance.

Applications could even use both APIs together, reading configuration to construct a Helidon builder and then adding to that builder OpenTelemetry objects created separately using the OpenTelemetry API.

The Helidon OpenTelemetry API Javadoc page lists the various types developers can use to prepare OpenTelemetry objects programmatically. As a starting point, the OpenTelemetryConfig interface and its Builder represents the top-level configuration for OpenTelemetry. Their Javadoc contains links to other types that compose the top-level object, and so on.

Later sections in this document also describe the configuration settings available.

The OpenTelemetry SDK documentation explains its API.

Note

Many applications do not need to use either the Helidon OpenTelemetry API or the OpenTelemetry API directly. They can instead rely completely on declarative Helidon configuration of OpenTelemetry.

Managing the Global OpenTelemetry Instance

Typically, an application uses the same OpenTelemetry instance throughout its execution. OpenTelemetry offers a global OpenTelemetry instance to make it easy for application code to set and obtain the global instance.

Similarly, the Helidon tracing API has a global Tracer.

In most cases, an application that prepares OpenTelemetry programmatically should initialize both of those by including code as shown in the following example.

Setting the global OpenTelemetry and Tracer instances in Helidon
import java.util.Map;
import io.helidon.telemetry.otelconfig.HelidonOpenTelemetry;
import io.opentelemetry.api.OpenTelemetry;

        // Application code using the OpenTelemetry API or the Helidon OpenTelemetry API or both.
        OpenTelemetry customOpenTelemetry = prepareOpenTelemetry();

        // App code to build any tags to be applied to every span.
        Map<String, String> tags = prepareTags();

        HelidonOpenTelemetry.global(customOpenTelemetry,
                                    "your-service-name",
                                    tags);
Copied

Assigning the Global Instance

Using Helidon to set the global OpenTelemetry instance has these effects:

  • Assigns the instance as the OpenTelemetry global instance.

  • Creates a Helidon Tracer using the OpenTelemetry instance and makes that the Helidon global Tracer.

Maven Coordinates

To enable various aspects of OpenTelemetry Support add one or more of the following dependencies to your project’s pom.xml (see Managing Dependencies).

Using the OpenTelemetry implementation of the Helidon Tracing API

Helidon offers an implementation of its neutral tracing API that uses OpenTelemetry. Add the following dependency to use OpenTelemetry for tracing.

Dependency to use the Helidon OpenTelemetry implementation of Helidon tracing
<dependency>
    <groupId>io.helidon.tracing.providers</groupId>
    <artifactId>helidon-tracing-providers-opentelemetry</artifactId>
    <scope>runtime</scope>
</dependency>
Copied

Adding OpenTelemetry Configuration and Builder Support

To allow deployers and end users to set up Helidon configuration to control OpenTelemetry behavior, add the following dependency.

Dependency to add Helidon OpenTelemetry config and programmatic builder support
<dependency>
    <groupId>io.helidon.telemetry</groupId>
    <artifactId>helidon-telemetry-opentelemetry-config</artifactId>
    <scope>runtime</scope> 
</dependency>
Copied
  • To use the Helidon OpenTelemetry API in your application code, remove this line or change it to <scope>compile</scope>.

Enabling Automatic Spans for HTTP Requests

Helidon’s tracing observability support automatically creates a new tracing span for each HTTP request if your project includes the following dependency.

Dependency for automatic HTTP request tracing
<dependency>
    <groupId>io.helidon.webserver.observe</groupId>
    <artifactId>helidon-webserver-observe-tracing</artifactId>
    <scope>runtime</scope>
</dependency>
Copied

By default, when Helidon SE creates spans automatically for HTTP requests, it uses a set of rules—​semantic conventions—​for choosing the span name and adding tags to each span.

OpenTelemetry prescribes its own semantic conventions. If you add the following dependency, Helidon follows the OpenTelemetry semantic conventions for spans instead of its own.

Dependency for Helidon support of the OpenTelemetry semantic conventions
<dependency>
    <groupId>io.helidon.webserver.observe</groupId>
    <artifactId>helidon-webserver-observe-telemetry-tracing</artifactId>
    <scope>runtime</scope>
</dependency>
Copied

Specifying Additional OpenTelemetry Dependencies

Most applications need to declare other runtime dependencies on OpenTelemetry artifacts because the configuration specifies—​or the application code uses—​particular OpenTelemetry types packaged in other artifacts. For example, OpenTelemetry packages various span exporters individually. (See this OpenTelemetry page on span exporters in particular.)

Configuration

You can control almost all of OpenTelemetry’s overall and tracing runtime behavior using Helidon configuration settings. Helidon constructs an OpenTelemetry object using the configuration. The resulting OpenTelemetry instance includes an OpenTelemetry TracerProvider if the config specifies settings for it under signals.tracing.

Controlling Overall OpenTelemetry Behavior

Several settings control the operation of OpenTelemetry as a whole, as shown in the next table.

Type: io.helidon.telemetry.otelconfig.HelidonOpenTelemetry

This is a standalone configuration type, prefix from configuration root: telemetry

Configuration options

Required configuration options
keytypedefault valuedescription
service

string

 

Service name used in sending telemetry data to the collector.

Optional configuration options
keytypedefault valuedescription
enabled

boolean

true

Whether the OpenTelemetry support is enabled.

global

boolean

true

Whether the io.opentelemetry.api.OpenTelemetry instance created from this configuration should be made the global one.

propagators

TextMapPropagator[]

new java.util.ArrayList<>(io.helidon.telemetry.otelconfig.ContextPropagationType.DEFAULT_PROPAGATORS)

OpenTelemetry io.opentelemetry.context.propagation.TextMapPropagator instances added explicitly by the app.

Default: tracecontext,baggage. See io.helidon.telemetry.otelconfig.ContextPropagationType

signals.tracing 

OpenTelemetry tracing settings.

Notes:

  • OpenTelemetry uses default propagators of tracecontext and baggage. (See the otel.propagators property in this OpenTelemetry guide.)

  • Setting global to true has the effect described in the section about global instances.

Controlling OpenTelemetry Tracing Behavior

The settings under settings.tracing prepare an OpenTelemetry TracerProvider. When your application uses the Helidon tracing API to obtain a Tracer, Helidon uses the TracerProvider prepared from this config to create the tracer.

The next table describes the OpenTelemetry tracing settings.

Type: io.helidon.telemetry.otelconfig.OpenTelemetryTracingConfig

Configuration options

Optional configuration options
keytypedefault valuedescription
attributes.booleans

Map<string, boolean>

 

Boolean attributes.

attributes.doubles

Map<string, double>

 

Double attributes.

attributes.longs

Map<string, long>

 

Long attributes.

attributes.strings

Map<string, string>

 

String attributes.

exporters

Map<string, SpanExporter>

 

Span exporters.

The key in the map is a unique name—​of the user’s choice—​for the exporter config settings. The SpanProcessorConfig.exporters() config setting for a processor config specifies zero or more of these names to associate the exporters built from the exporter configs with the processor built from the processor config.

processors 

Settings for span processors.

sampler

Sampler

 

Tracing sampler.

span-limits

SpanLimits

 

Tracing span limits.

OpenTelemetry applies the defaults described in the next table.

Default tracing settings applied by OpenTelemetry
SettingOpenTelemetry default (and OpenTelemetry doc link)
exportersotlp with grpc protocol - see "Properties: exporters, otel.traces.exporter property"
processorsbatch with defaults - see "Properties for batch span processor(s)"
samplerparentbased_always_on - see "Properties for sampler"
span-limitsSee tracing "Properties for span limits"

Sections below describe how to set up the tracing signal configuration:

Assigning Attributes

Configured attributes are key/value pairs that OpenTelemetry attaches to each tracing span. OpenTelemetry supports attributes of type String, long, double, and boolean. The configuration stricture groups attributes by type so Helidon knows precisely what type you intend for each attribute.

Associating Exporters and Processors

Ultimately, OpenTelemetry transmits the telemetry data it gathers to a backend system—​such as Grafana, Prometheus, Jaeger, or others—​where you can view and query the data. OpenTelemetry tracing first processes each data observation within the JVM and then exports the processed data to one or more backends. Each span processor uses one or more span exporters to transmit telemetry data.

You link processors and exporters in config selectively as follows:

  • Name each exporter if you have more than one.

  • List, for each processor, the names of exporters it should use. Omitting a processor’s list of exporter names causes the processor to use all configured exporters.

The following examples show increasingly-complicated scenarios:

  • Default

  • Minimal configuration

  • Maximum flexibility

For many applications the default and minimal scenarios work well.

Default

This scenario includes no configuration at all for either processors or exporters.

OpenTelemetry uses its default processor (batch) with its default exporter (otlp using grpc). |

telemetry:
  service: "inventory"
  tracing:
    sampler: "always_off"
Copied
Minimal configuration

The user configures at most one processor and at most one exporter.

The single processor uses the single exporter.

No exporter name is declared or referenced.

telemetry:
  service: "inventory"
  tracing:
    sampler: "always_off"
    exporters:
      - type: zipkin
        compression: gzip
    processors:
      - type: batch
        max-queue-size: 50
Copied
Maximum flexibility

The user configures possibly multiple processors and possibly multiple named exporters. Each processor’s configuration lists the names of the exporters it should use; no names means all exporters.

The first processor (type batch) uses both exporters because it does not specify any exporter names. The second processor uses only the alternate-otlp exporter.|

telemetry:
  service: "inventory"
  tracing:
    sampler: "always_off"
    exporters:
      - type: zipkin
        compression: gzip
        name: "compressed-zipkin"
      - endpoint: "http://collect.com:4317"
        name: "alternate-otlp""
    processors:
      - type: batch
        max-queue-size: 50
      - type: simple
        exporters: ["alternate-otlp"]
Copied

Configuring Span Processors

An OpenTelemetry span processor is one of the following types:

  • simple - The processor sends each telemetry observation to its exporters for transmission as soon as it receives the observation.

  • batch - The processor groups observations into batches and sends a batch at a time to its exporters for transmission.

In the table below only the type and exporters setting apply to simple span processors; the other settings are for batch processors.

Type: io.helidon.telemetry.otelconfig.BatchSpanProcessorConfig

Configuration options
Required configuration options
keytypedefault valuedescription
type

SpanProcessorType (SIMPLE, BATCH)

 

Span processor type.

Allowed values:

  • SIMPLE: Simple Span Processor.

  • BATCH: Batch Span Processor.

Optional configuration options
keytypedefault valuedescription
exporters

string[]

 

Name(s) of the span exporter(s) this span processor should use; specifying no names uses all configured exporters (or if no exporters are configured, the default OpenTelemetry exporter(s)).

Each name must be the name of one of the configured OpenTelemetryTracingConfig.exporterConfigs().

max-export-batch-size

int

 

Maximum number of spans batched for export together. OpenTelemetry requires this value to not exceed the maxQueueSize().

max-queue-size

int

 

Maximum number of spans retained before discarding excess unexported ones.

schedule-delay

Duration

 

Delay between consecutive exports.

timeout

Duration

 

Maximum time an export can run before being cancelled.

Configuring Span Exporters

OpenTelemetry supports many types of exporters, each with its own configurable settings. Helidon configuration supports several of the most popular exporters, listed in the table below.

Note

You need to add dependencies to your project for the exporters your application uses, even ones supported by Helidon config.

If you need to use an exporter that is not in the table:

  • Add a dependency on the OpenTelemetry artifact that contains that exporter type.

  • Add application code that prepares the exporter instance.

  • Prepare the Helidon OpenTelemetry builders programmatically and add your exporter instance to the builder.

Exporter typeOpenTelemetry Java TypeDependency to add - see also the OpenTelemetry documentation
otlp
(see protocol setting below)
OtlpGrpcSpanExporter
<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
Copied
OtlpHttpSpanExporter
zipkinZipkinSpanExporter
<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-exporter-zipkin</artifactId>
</dependency>
Copied
consoleLoggingSpanExporter

included

logging_otlpOtlpJsonLoggingSpanExporter

included

Configuring an OTLP Exporter

The configuration for OTLP exporters is the same, regardless of which protocol you choose: grpc (the default) or http/proto. OpenTelemetry applies some different default values depending on the protocol.

Type: io.helidon.telemetry.otelconfig.OtlpExporterConfig

Configuration options

Optional configuration options
keytypedefault valuedescription
certificate 

Trusted certificates.

client.certificate 

TLS certificate.

client.key 

TLS client key.

compression

CompressionType (GZIP, NONE)

 

Compression the exporter uses.

Allowed values:

  • GZIP: GZIP compression.

  • NONE: No compression.

endpoint

URI

 

Endpoint of the collector to which the exporter should transmit.

headers

Map<string, string>

 

Headers added to each export message.

protocol

OtlpExporterProtocolType (HTTP_PROTO, GRPC)

OtlpExporterProtocolType.DEFAULT

Exporter protocol type.

Allowed values:

  • HTTP_PROTO: http/proto protocol type.

  • GRPC: grpc protocol type.

retry-policy

RetryPolicy

 

Retry policy.

timeout

Duration

 

Exporter timeout.

The OpenTelemetry documentation describes the defaults; see the "Properties for `otlp `span, metric, and log exporters" section there.

OpenTelemetry defaults for tracing
SettingOpenTelemetry default (and OpenTelemetry doc link)
compressionnone
endpointgrpc protocol: http://localhost:4317http/proto protocol: http://localhost:4318
protocolgrpc
retry-policynone
timeout10 seconds
Configuring a Zipkin Exporter

Configuration options

Optional configuration options
keytypedefault valuedescription
compression

CompressionType (GZIP, NONE)

 

Compression type.

Allowed values:

  • GZIP: GZIP compression.

  • NONE: No compression.

encoder

SpanBytesEncoder (JSON_V1, THRIFT, JSON_V2, PROTO3)

 

Encoder type.

endpoint

URI

 

Collector endpoint to which this exporter should transmit.

timeout

Duration

 

Exporter timeout.

The OpenTelemetry documentation describes the defaults; see the "Properties for Zipkin span exporters" section there.

OpenTelemetry defaults for Zipkin exporters
SettingOpenTelemetry default value
compressionnone
encoderJSON_V2
endpointhttp://localhost:9411/api/v2/spans
timeout10 seconds

Configuring the Span Sampler

OpenTelemetry offers different ways of sampling data—​deciding which tracing spans tp capture and send to the backend. The OpenTelemetry documentation describes sampling in more detail.

Helidon configuration supports the sampler implementations that reside in the opentelemetry-sdk as listed in the table below. Other samplers are in other components. If you need to use one of those:

  • Add the relevant OpenTelemetry dependency to your project.

  • Instantiate the span sample you need.

  • Prepare the sampler and the OpenTelemetry-related builders programmatically and use your sampler to assign the sampler the OpenTelemetryTracer.Builder should use.

Type: io.helidon.telemetry.otelconfig.SamplerConfig

Configuration options

Optional configuration options
keytypedefault valuedescription
param

double

 

Sampler parameter.

type

SamplerType (ALWAYS_ON, ALWAYS_OFF, TRACEIDRATIO, PARENTBASED_ALWAYS_ON, PARENTBASED_ALWAYS_OFF, PARENTBASED_TRACEIDRATIO)

SamplerType.DEFAULT

Sampler type.

Allowed values:

  • ALWAYS_ON: Always on sampler.

  • ALWAYS_OFF: Always off sampler.

  • TRACEIDRATIO: Trace ID ratio-based sampler.

  • PARENTBASED_ALWAYS_ON: Parent-based always-on sampler.

  • PARENTBASED_ALWAYS_OFF: Parent-based always-off sampler.

  • PARENTBASED_TRACEIDRATIO: Parent-based trace ID ration-based sampler.

Configuring Span Limits

OpenTelemetry allows you to constrain certain aspects of the data it gathers in tracing spans. By assigning the settings in the table below, you can apply the span limits you want.

Type: io.helidon.telemetry.otelconfig.SpanLimitsConfig

Configuration options

Optional configuration options
keytypedefault valuedescription
max-attribute-value-length

int

 

Maximum attribute value length.

max-attributes

int

 

Maximum number of attributes.

max-attributes-per-event

int

 

Maximum number of attributes per event.

max-attributes-per-link

int

 

Maximum number of attributes per link.

max-events

int

 

Maximum number of events.

max-links

int

 

Maximum number of links.

The OpenTelemetry documentation describes the defaults; see the "Properties for span limits" section there.

OpenTelemetry defaults for span limits
SettingOpenTelemetry Default
max-attribute-value-lengthno limit
max-attributes128
max-attributes-per-event128
max-events128
max-links128

Additional Information