Overview
Helidon SE support for OpenTelemetry configuration and semantic conventions as described below is currently an incubating feature. We intend to support it going forward, but we might withdraw the feature entirely or change its external API and behavior in backward-incompatible ways across dot releases.
Helidon SE supports OpenTelemetry in several important ways:
Implements the neutral Helidon tracing API using OpenTelemetry
Allows users to assign OpenTelemetry settings as follows:
Declaratively, using Helidon config under the top-level
telemetryconfig keyProgrammatically, using the OpenTelemetry SDK API and the Helidon OpenTelemetry API
Conforms to the OpenTelemetry semantic conventions for automatically-created spans
OpenTelemetry models observability as a set of signals. Each signal—for example metrics, tracing, and logging—is an origin of monitoring data, and each has configurable settings which control its behavior.
Helidon’s config support for OpenTelemetry has certain config attributes which apply to OpenTelemetry as a whole, others which pertain to individual signals, and still more which describe lower-level elements within a signal.
The Helidon OpenTelemetry configuration format, the Helidon OpenTelemetry API, and this documentation all follow this hierarchy:
This document describes how to configure each level in the hierarchy and covers general topics related to Helidon’s support of OpenTelemetry.
API
There are two APIs that might be useful to developers working with OpenTelemetry:
The Helidon OpenTelemetry API - useful for mapping configuration sources to Helidon builders and, ultimately, OpenTelemetry objects.
The OpenTelemetry API - useful for creating OpenTelemetry objects apart from Helidon configuration sources.
The types of the Helidon OpenTelemetry API correspond closely to the configuration structures described in later sections of this document. Application code can use Helidon OpenTelemetry builders to prepare and construct each of the configurable entities to ultimately prepare an OpenTelemetry instance set up according to the application’s needs.
That said, application code can equally well use the OpenTelemetry API and its builders to prepare the OpenTelemetry instance.
Applications could even use both APIs together, reading configuration to construct a Helidon builder and then adding to that builder OpenTelemetry objects created separately using the OpenTelemetry API.
The Helidon OpenTelemetry API Javadoc page lists the various types developers can use to prepare OpenTelemetry objects programmatically. As a starting point, the OpenTelemetryConfig interface and its Builder represents the top-level configuration for OpenTelemetry. Their Javadoc contains links to other types that compose the top-level object, and so on.
Later sections in this document also describe the configuration settings available.
The OpenTelemetry SDK documentation explains its API.
Note
Many applications do not need to use either the Helidon OpenTelemetry API or the OpenTelemetry API directly. They can instead rely completely on declarative Helidon configuration of OpenTelemetry.
Managing the Global OpenTelemetry Instance
Typically, an application uses the same OpenTelemetry instance throughout its execution. OpenTelemetry offers a global OpenTelemetry instance to make it easy for application code to set and obtain the global instance.
Similarly, the Helidon tracing API has a global Tracer.
In most cases, an application that prepares OpenTelemetry programmatically should initialize both of those by including code as shown in the following example.
OpenTelemetry and Tracer instances in Helidonimport java.util.Map;
import io.helidon.telemetry.otelconfig.HelidonOpenTelemetry;
import io.opentelemetry.api.OpenTelemetry;
// Application code using the OpenTelemetry API or the Helidon OpenTelemetry API or both.
OpenTelemetry customOpenTelemetry = prepareOpenTelemetry();
// App code to build any tags to be applied to every span.
Map<String, String> tags = prepareTags();
HelidonOpenTelemetry.global(customOpenTelemetry,
"your-service-name",
tags);Assigning the Global Instance
Using Helidon to set the global OpenTelemetry instance has these effects:
Assigns the instance as the OpenTelemetry global instance.
Creates a Helidon
Tracerusing the OpenTelemetry instance and makes that the Helidon globalTracer.
Maven Coordinates
To enable various aspects of OpenTelemetry Support add one or more of the following dependencies to your project’s pom.xml (see Managing Dependencies).
Using the OpenTelemetry implementation of the Helidon Tracing API
Helidon offers an implementation of its neutral tracing API that uses OpenTelemetry. Add the following dependency to use OpenTelemetry for tracing.
<dependency>
<groupId>io.helidon.tracing.providers</groupId>
<artifactId>helidon-tracing-providers-opentelemetry</artifactId>
<scope>runtime</scope>
</dependency>Adding OpenTelemetry Configuration and Builder Support
To allow deployers and end users to set up Helidon configuration to control OpenTelemetry behavior, add the following dependency.
<dependency>
<groupId>io.helidon.telemetry</groupId>
<artifactId>helidon-telemetry-opentelemetry-config</artifactId>
<scope>runtime</scope>
</dependency>- To use the Helidon OpenTelemetry API in your application code, remove this line or change it to
<scope>compile</scope>.
Enabling Automatic Spans for HTTP Requests
Helidon’s tracing observability support automatically creates a new tracing span for each HTTP request if your project includes the following dependency.
<dependency>
<groupId>io.helidon.webserver.observe</groupId>
<artifactId>helidon-webserver-observe-tracing</artifactId>
<scope>runtime</scope>
</dependency>By default, when Helidon SE creates spans automatically for HTTP requests, it uses a set of rules—semantic conventions—for choosing the span name and adding tags to each span.
OpenTelemetry prescribes its own semantic conventions. If you add the following dependency, Helidon follows the OpenTelemetry semantic conventions for spans instead of its own.
<dependency>
<groupId>io.helidon.webserver.observe</groupId>
<artifactId>helidon-webserver-observe-telemetry-tracing</artifactId>
<scope>runtime</scope>
</dependency>Specifying Additional OpenTelemetry Dependencies
Most applications need to declare other runtime dependencies on OpenTelemetry artifacts because the configuration specifies—or the application code uses—particular OpenTelemetry types packaged in other artifacts. For example, OpenTelemetry packages various span exporters individually. (See this OpenTelemetry page on span exporters in particular.)
Configuration
You can control almost all of OpenTelemetry’s overall and tracing runtime behavior using Helidon configuration settings. Helidon constructs an OpenTelemetry object using the configuration. The resulting OpenTelemetry instance includes an OpenTelemetry TracerProvider if the config specifies settings for it under signals.tracing.
Controlling Overall OpenTelemetry Behavior
Several settings control the operation of OpenTelemetry as a whole, as shown in the next table.
Type: io.helidon.telemetry.otelconfig.HelidonOpenTelemetry
This is a standalone configuration type, prefix from configuration root: telemetry
Configuration options
| key | type | default value | description |
|---|---|---|---|
service | string | Service name used in sending telemetry data to the collector. |
| key | type | default value | description |
|---|---|---|---|
enabled | boolean | true | Whether the OpenTelemetry support is enabled. |
global | boolean | true | Whether the io.opentelemetry.api.OpenTelemetry instance created from this configuration should be made the global one. |
propagators | TextMapPropagator[] | new java.util.ArrayList<>(io.helidon.telemetry.otelconfig.ContextPropagationType.DEFAULT_PROPAGATORS) | OpenTelemetry io.opentelemetry.context.propagation.TextMapPropagator instances added explicitly by the app. Default: |
signals.tracing | OpenTelemetry tracing settings. |
Notes:
OpenTelemetry uses default propagators of
tracecontextandbaggage. (See theotel.propagatorsproperty in this OpenTelemetry guide.)Setting
globaltotruehas the effect described in the section about global instances.
Controlling OpenTelemetry Tracing Behavior
The settings under settings.tracing prepare an OpenTelemetry TracerProvider. When your application uses the Helidon tracing API to obtain a Tracer, Helidon uses the TracerProvider prepared from this config to create the tracer.
The next table describes the OpenTelemetry tracing settings.
Type: io.helidon.telemetry.otelconfig.OpenTelemetryTracingConfig
Configuration options
| key | type | default value | description |
|---|---|---|---|
attributes.booleans | Map<string, boolean> | Boolean attributes. | |
attributes.doubles | Map<string, double> | Double attributes. | |
attributes.longs | Map<string, long> | Long attributes. | |
attributes.strings | Map<string, string> | String attributes. | |
exporters | Map<string, SpanExporter> | Span exporters. The key in the map is a unique name—of the user’s choice—for the exporter config settings. The SpanProcessorConfig.exporters() config setting for a processor config specifies zero or more of these names to associate the exporters built from the exporter configs with the processor built from the processor config. | |
processors | Settings for span processors. | ||
sampler | Sampler | Tracing sampler. | |
span-limits | SpanLimits | Tracing span limits. |
OpenTelemetry applies the defaults described in the next table.
| Setting | OpenTelemetry default (and OpenTelemetry doc link) |
|---|---|
exporters | otlp with grpc protocol - see "Properties: exporters, otel.traces.exporter property" |
processors | batch with defaults - see "Properties for batch span processor(s)" |
sampler | parentbased_always_on - see "Properties for sampler" |
span-limits | See tracing "Properties for span limits" |
Sections below describe how to set up the tracing signal configuration:
Assigning Attributes
Configured attributes are key/value pairs that OpenTelemetry attaches to each tracing span. OpenTelemetry supports attributes of type String, long, double, and boolean. The configuration stricture groups attributes by type so Helidon knows precisely what type you intend for each attribute.
Associating Exporters and Processors
Ultimately, OpenTelemetry transmits the telemetry data it gathers to a backend system—such as Grafana, Prometheus, Jaeger, or others—where you can view and query the data. OpenTelemetry tracing first processes each data observation within the JVM and then exports the processed data to one or more backends. Each span processor uses one or more span exporters to transmit telemetry data.
You link processors and exporters in config selectively as follows:
Name each exporter if you have more than one.
List, for each processor, the names of exporters it should use. Omitting a processor’s list of exporter names causes the processor to use all configured exporters.
The following examples show increasingly-complicated scenarios:
Default
Minimal configuration
Maximum flexibility
For many applications the default and minimal scenarios work well.
Default
This scenario includes no configuration at all for either processors or exporters.
OpenTelemetry uses its default processor (batch) with its default exporter (otlp using grpc). |
telemetry:
service: "inventory"
tracing:
sampler: "always_off"Minimal configuration
The user configures at most one processor and at most one exporter.
The single processor uses the single exporter.
No exporter name is declared or referenced.
telemetry:
service: "inventory"
tracing:
sampler: "always_off"
exporters:
- type: zipkin
compression: gzip
processors:
- type: batch
max-queue-size: 50Maximum flexibility
The user configures possibly multiple processors and possibly multiple named exporters. Each processor’s configuration lists the names of the exporters it should use; no names means all exporters.
The first processor (type batch) uses both exporters because it does not specify any exporter names. The second processor uses only the alternate-otlp exporter.|
telemetry:
service: "inventory"
tracing:
sampler: "always_off"
exporters:
- type: zipkin
compression: gzip
name: "compressed-zipkin"
- endpoint: "http://collect.com:4317"
name: "alternate-otlp""
processors:
- type: batch
max-queue-size: 50
- type: simple
exporters: ["alternate-otlp"]Configuring Span Processors
An OpenTelemetry span processor is one of the following types:
simple - The processor sends each telemetry observation to its exporters for transmission as soon as it receives the observation.
batch - The processor groups observations into batches and sends a batch at a time to its exporters for transmission.
In the table below only the type and exporters setting apply to simple span processors; the other settings are for batch processors.
Type: io.helidon.telemetry.otelconfig.BatchSpanProcessorConfig
Configuration options
| key | type | default value | description |
|---|---|---|---|
type | SpanProcessorType (SIMPLE, BATCH) | Span processor type. Allowed values:
|
| key | type | default value | description |
|---|---|---|---|
exporters | string[] | Name(s) of the span exporter(s) this span processor should use; specifying no names uses all configured exporters (or if no exporters are configured, the default OpenTelemetry exporter(s)). Each name must be the name of one of the configured OpenTelemetryTracingConfig.exporterConfigs(). | |
max-export-batch-size | int | Maximum number of spans batched for export together. OpenTelemetry requires this value to not exceed the maxQueueSize(). | |
max-queue-size | int | Maximum number of spans retained before discarding excess unexported ones. | |
schedule-delay | Duration | Delay between consecutive exports. | |
timeout | Duration | Maximum time an export can run before being cancelled. |
Configuring Span Exporters
OpenTelemetry supports many types of exporters, each with its own configurable settings. Helidon configuration supports several of the most popular exporters, listed in the table below.
Note
You need to add dependencies to your project for the exporters your application uses, even ones supported by Helidon config.
If you need to use an exporter that is not in the table:
Add a dependency on the OpenTelemetry artifact that contains that exporter type.
Add application code that prepares the exporter instance.
Prepare the Helidon OpenTelemetry builders programmatically and add your exporter instance to the builder.
| Exporter type | OpenTelemetry Java Type | Dependency to add - see also the OpenTelemetry documentation |
|---|---|---|
otlp(see protocol setting below) | OtlpGrpcSpanExporter | content_copy |
OtlpHttpSpanExporter | ||
zipkin | ZipkinSpanExporter | content_copy |
console | LoggingSpanExporter | included |
logging_otlp | OtlpJsonLoggingSpanExporter | included |
Configuring an OTLP Exporter
The configuration for OTLP exporters is the same, regardless of which protocol you choose: grpc (the default) or http/proto. OpenTelemetry applies some different default values depending on the protocol.
Configuration options
| key | type | default value | description |
|---|---|---|---|
certificate | Trusted certificates. | ||
client.certificate | TLS certificate. | ||
client.key | TLS client key. | ||
compression | CompressionType (GZIP, NONE) | Compression the exporter uses. Allowed values:
| |
endpoint | URI | Endpoint of the collector to which the exporter should transmit. | |
headers | Map<string, string> | Headers added to each export message. | |
protocol | OtlpExporterProtocolType (HTTP_PROTO, GRPC) | OtlpExporterProtocolType.DEFAULT | Exporter protocol type. Allowed values:
|
retry-policy | RetryPolicy | Retry policy. | |
timeout | Duration | Exporter timeout. |
The OpenTelemetry documentation describes the defaults; see the "Properties for `otlp `span, metric, and log exporters" section there.
| Setting | OpenTelemetry default (and OpenTelemetry doc link) | |
|---|---|---|
compression | none | |
endpoint | grpc protocol: http://localhost:4317 | http/proto protocol: http://localhost:4318 |
protocol | grpc | |
retry-policy | none | |
timeout | 10 seconds |
Configuring a Zipkin Exporter
Configuration options
| key | type | default value | description |
|---|---|---|---|
compression | CompressionType (GZIP, NONE) | Compression type. Allowed values:
| |
encoder | SpanBytesEncoder (JSON_V1, THRIFT, JSON_V2, PROTO3) | Encoder type. | |
endpoint | URI | Collector endpoint to which this exporter should transmit. | |
timeout | Duration | Exporter timeout. |
The OpenTelemetry documentation describes the defaults; see the "Properties for Zipkin span exporters" section there.
| Setting | OpenTelemetry default value |
|---|---|
compression | none |
encoder | JSON_V2 |
endpoint | http://localhost:9411/api/v2/spans |
timeout | 10 seconds |
Configuring the Span Sampler
OpenTelemetry offers different ways of sampling data—deciding which tracing spans tp capture and send to the backend. The OpenTelemetry documentation describes sampling in more detail.
Helidon configuration supports the sampler implementations that reside in the opentelemetry-sdk as listed in the table below. Other samplers are in other components. If you need to use one of those:
Add the relevant OpenTelemetry dependency to your project.
Instantiate the span sample you need.
Prepare the sampler and the OpenTelemetry-related builders programmatically and use your sampler to assign the sampler the
OpenTelemetryTracer.Buildershould use.
Configuration options
| key | type | default value | description |
|---|---|---|---|
param | double | Sampler parameter. | |
type | SamplerType (ALWAYS_ON, ALWAYS_OFF, TRACEIDRATIO, PARENTBASED_ALWAYS_ON, PARENTBASED_ALWAYS_OFF, PARENTBASED_TRACEIDRATIO) | SamplerType.DEFAULT | Sampler type. Allowed values:
|
Configuring Span Limits
OpenTelemetry allows you to constrain certain aspects of the data it gathers in tracing spans. By assigning the settings in the table below, you can apply the span limits you want.
Configuration options
| key | type | default value | description |
|---|---|---|---|
max-attribute-value-length | int | Maximum attribute value length. | |
max-attributes | int | Maximum number of attributes. | |
max-attributes-per-event | int | Maximum number of attributes per event. | |
max-attributes-per-link | int | Maximum number of attributes per link. | |
max-events | int | Maximum number of events. | |
max-links | int | Maximum number of links. |
The OpenTelemetry documentation describes the defaults; see the "Properties for span limits" section there.
| Setting | OpenTelemetry Default |
|---|---|
max-attribute-value-length | no limit |
max-attributes | 128 |
max-attributes-per-event | 128 |
max-events | 128 |
max-links | 128 |