OpenTelemetry
Overview
Helidon SE supports OpenTelemetry in several important ways:
- Implements the neutral Helidon tracing API using OpenTelemetry
- Allows users to assign OpenTelemetry settings as follows:
- Declaratively, using Helidon config under the top-level
telemetryconfig key - Programmatically, using the OpenTelemetry SDK API and the Helidon OpenTelemetry API
- Declaratively, using Helidon config under the top-level
- Conforms to the OpenTelemetry semantic conventions for automatically-created spans and metrics for HTTP requests
- Allows publishing Helidon metrics to backend systems using OTLP.
OpenTelemetry models observability as a set of signals.
Each signal—for example metrics, tracing, and logging—is an origin of monitoring data, and each has configurable settings which control its behavior.
Helidon’s config support for OpenTelemetry has certain config attributes which apply to OpenTelemetry as a whole, others which pertain to individual signals, and still more which describe lower-level elements within a signal.
The Helidon OpenTelemetry configuration format, the Helidon OpenTelemetry API, and this documentation all follow this hierarchy:
This document describes how to configure each level in the hierarchy and covers general topics related to Helidon’s support of OpenTelemetry.
API
There are two APIs that might be useful to developers working with OpenTelemetry:
- The Helidon OpenTelemetry API - useful for mapping configuration sources to Helidon builders and, ultimately, OpenTelemetry objects.
- The OpenTelemetry API - useful for creating OpenTelemetry objects apart from Helidon configuration sources.
The types in the Helidon OpenTelemetry API correspond closely to the
configuration structures described in later sections of this document.
Application code can use Helidon OpenTelemetry builders to prepare and construct
each of the configurable entities to ultimately prepare an OpenTelemetry
instance set up according to the application’s needs.
That said, application code can equally well use the OpenTelemetry API and its
builders to prepare the OpenTelemetry instance.
Applications could even use both APIs together, reading configuration to construct a Helidon builder and then adding to that builder OpenTelemetry objects created separately using the OpenTelemetry API.
The Helidon OpenTelemetry API Javadoc page lists the various
types developers can use to prepare OpenTelemetry objects programmatically. As a
starting point, the OpenTelemetryConfig interface and its
Builder represents the top-level configuration for OpenTelemetry.
Their Javadoc contains links to other types that compose the top-level object,
and so on.
Later sections in this document also describe the configuration settings available.
The OpenTelemetry SDK documentation explains its API.
Global Instance
Typically, an application uses the same OpenTelemetry instance throughout its
execution. OpenTelemetry offers a global OpenTelemetry instance to make it
easy for application code to set and obtain the global instance.
Similarly, the Helidon tracing API has a global Tracer.
In most cases, an application that prepares OpenTelemetry programmatically should initialize both of those by including code as shown in the following example.
Setting the global OpenTelemetry and Tracer instances in Helidon:
import java.util.Map;
import io.helidon.telemetry.otelconfig.HelidonOpenTelemetry;
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.context.Scope;
import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
// Application code using the OpenTelemetry API or the Helidon OpenTelemetry API or both.
OpenTelemetry customOpenTelemetry = prepareOpenTelemetry();
// App code to build any tags to be applied to every span.
Map<String, String> tags = prepareTags();
HelidonOpenTelemetry.global(customOpenTelemetry, "your-service-name", tags);
Assigning the Global Instance
Using Helidon to set the global OpenTelemetry instance has these effects:
- Assigns the instance as the OpenTelemetry global instance.
- Creates a Helidon
Tracerusing the OpenTelemetry instance and makes that the Helidon globalTracer.
Services.get(Tracer.class) instead of Tracer.global().Maven Coordinates
To enable various aspects of OpenTelemetry Support add one or more of the
following dependencies to your project’s pom.xml (see Managing
Dependencies).
Helidon Tracing provider:
<dependency>
<groupId>io.helidon.tracing.providers</groupId>
<artifactId>helidon-tracing-providers-opentelemetry</artifactId>
</dependency>
Helidon Config and programmatic builder support:
<dependency>
<groupId>io.helidon.telemetry</groupId>
<artifactId>helidon-telemetry-opentelemetry-config</artifactId>
</dependency>
Helidon WebServer OpenTelemetry Tracing semantics:
<dependency>
<groupId>io.helidon.webserver.observe</groupId>
<artifactId>helidon-webserver-observe-telemetry-tracing</artifactId>
<scope>runtime</scope>
</dependency>
Helidon WebServer OpenTelemetry Metrics semantics:
<dependency>
<groupId>io.helidon.webserver.observe</groupId>
<artifactId>helidon-webserver-observe-telemetry-metrics</artifactId>
<scope>runtime</scope>
</dependency>
WebClient Support
Helidon supports the OpenTelemetry semantic conventions for outgoing traffic which uses the Helidon WebClient. See the Helidon WebClient documentation.
Additional Dependencies
Most applications need to declare other runtime dependencies on OpenTelemetry artifacts because the configuration specifies—or the application code uses—particular OpenTelemetry types packaged in other artifacts.
For example, OpenTelemetry exporters are packaged individually or as related groups. See this section below for some specific dependencies to consider adding for particular exporters.
These exporters transmit telemetry data using a different protocol. (See this OpenTelemetry page.)
Configuration options
| Key | Type | Default | Description |
|---|---|---|---|
enabled | Boolean | true | Whether the OpenTelemetry support is enabled |
global | Boolean | true | Whether the io. instance created from this configuration should be made the global one |
propagators | List< | OpenTelemetry io. instances added explicitly by the app | |
service | String | Service name used in sending telemetry data to the collector | |
signals | Configuration for signals |
Notes:
- OpenTelemetry uses default propagators of
tracecontextandbaggage. (See theotel.propagatorsproperty in this OpenTelemetry guide.) - Setting
globaltotruehas the effect described in the section about global instances.
Signal Attributes
Attributes are key/value pairs that OpenTelemetry attaches to each transmission of a signal.
OpenTelemetry supports attributes of type String, long, double, and
boolean.
The Helidon configuration structure groups attributes by type so Helidon can indicate precisely to OpenTelemetry what type you intend for each attribute.
Configuration options
You can add attributes to the configuration for any of the signals under the
signal’s attributes section.
| Key | Type | Description |
|---|---|---|
longs | Map< | Long attributes |
strings | Map< | String attributes |
doubles | Map< | Double attributes |
booleans | Map< | Boolean attributes |
The following example shows attribute settings for the tracing signal.
Configuration Example
telemetry:
service: my-helidon-service
tracing:
attributes:
strings:
attr1: 12
attr5: "any old thing"
attr7: something
longs:
attr2: 12
doubles:
attr3: 24.5
attr6: 12
booleans:
attr4: true
Processors & Readers
Two types of processors and readers are provided:
simple: sends observations upon receivebatch: groups observations into batches
In the table below only the type and exporters setting apply to simple
processors; the other settings are for batch processors.
Configuration options
| Key | Type | Description |
|---|---|---|
schedule- | Duration | Delay between consecutive exports |
max- | Integer | Maximum number of items batched for export together |
type | Processor | Processor type |
max- | Integer | Maximum number of items retained before discarding excess unexported ones |
timeout | Duration | Maximum time an export can run before being cancelled |
exporters | List< | Name(s) of the exporter(s) this processor should use; specifying no names uses all configured exporters (or if no exporters are configured, the default OpenTelemetry exporter(s)) |
Exporters
Exporter objects in OpenTelemetry are specific to both how they transmit telemetry data and what signal they work with. Even so, many exporter settings are very similar across different signals. This section describes the behavior and configuration that is common among exporters. Refer to the sections below that describe each signal to see what additional exporter settings, if any, each signal adds.
Helidon configuration supports several of the most popular exporters, discussed below.
Configuration options
| Key | Type | Default | Description |
|---|---|---|---|
connect- | Duration | Connection timeout | |
headers | Map< | Headers added to each export message | |
endpoint | URI | Endpoint of the collector to which the exporter should transmit | |
memory- | Memory | Memory mode | |
protocol | String | DEFAULT | Exporter protocol type |
internal- | Internal | Self-monitoring telemetry OpenTelemetry should collect | |
certificate | Resource | Trusted certificates | |
client. | Resource | TLS client key | |
client. | Resource | TLS certificate | |
compression | Compression | Compression the exporter uses | |
retry- | Retry | Retry policy | |
timeout | Duration | Exporter timeout |
Common Configuration for OTLP exporters:
| Setting | Protocol | OpenTelemetry default |
|---|---|---|
compression | none | |
endpoint | grpc | http://localhost:4317 |
http/ | http://localhost:4318 | |
protocol | grpc | |
retry-policy | none | |
timeout | 10 seconds |
OTLP Retry Policy
You can control how each exporter retries if a transmission to a backend fails.
Configuration options
| Key | Type | Description |
|---|---|---|
max- | Duration | Maximum backoff time |
initial- | Duration | Initial backoff time |
max- | Double | Maximum backoff multiplier |
max- | Integer | Maximum number of retry attempts |
OpenTelemetry also supports a Zipkin exporter which it has recently deprecated.
Zipkin Exporter
Configuration options
| Key | Type | Description |
|---|---|---|
endpoint | URI | Collector endpoint to which this exporter should transmit |
compression | Compression | Compression type |
encoder | Span | Encoder type |
timeout | Duration | Exporter timeout |
Configuration for Zipkin exporters
The OpenTelemetry documentation describes the defaults; see the "Properties for Zipkin span exporters" section there.
OpenTelemetry defaults for Zipkin exporters:
| Setting | OpenTelemetry default value |
|---|---|
compression | none |
encoder | JSON_V2 |
endpoint | http://localhost:9411/api/v2/spans |
timeout | 10 seconds |
OpenTelemetry provides other exporters, often used for debugging:
console
Writes telemetry data at theINFOlevel using thejava.util.logging.Loggerforio.opentelemetry.exporter.logging.Logging{signal}Expoerterlogging-otlp
Writes telemetry data in JSON format to the logger for the particular OpenTelemetry implementation class (e.g.,OtlpJsonLoggingMetricExporter).
The console and logging-otlp have no configuration that is common across all
signals.
The table below describes the exporter types that Helidon configuration supports and what dependency your project needs to support them.
If you need to use an exporter that is not in the table:
- Add a dependency on the OpenTelemetry artifact that contains that exporter type.
- Add application code that prepares the exporter instance.
- Prepare the Helidon OpenTelemetry builders programmatically and add your exporter instance to the builder.
In the table below, the Maven artifacts are all in the io.opentelemetry group.
| Exporter type | OpenTelemetry Java Type | Artifact ID to add - seealso the OpenTelemetry documentation |
|---|---|---|
otlp | OtlpGrpc{signal}Exporter | opentelemetry-exporter-otlp |
OtlpHttp{signal}Exporter | ||
zipkin | ZipkinSpanExporter | opentelemetry-exporter-zipkin |
console | Logging{signal}Exporter | opentelemetry-exporter-logging |
logging_otlp | OtlpJsonLogging{signal}ExporterSystemOutLogRecordExporter |
Assigning Exporters
In configuration, you link processors and readers with the exporters you want each to use as follows:
- For clarity, name each exporter if you have more than one.
- Optionally specify for each processor or reader the names of the exporters it
should use.
If you omit the exporter names for a processor or reader, Helidon associates it with all configured exporters. If you configure no exporters explicitly, Helidon associates the OpenTelemetry default exporter with the processor or reader.
The following examples show increasingly-complicated scenarios using tracing as the signal:
- Default
- Minimal configuration
- Maximum flexibility
For many applications the default and minimal scenarios work well.
Default
This scenario includes no configuration at all for either processors or exporters.
OpenTelemetry uses its default processor (batch) with its default exporter
(otlp using grpc). |
telemetry:
service: "inventory"
tracing:
sampler: "always_off"
Minimal configuration
The user configures at most one processor and at most one exporter.
The single processor uses the single exporter.
No exporter name is declared or referenced.
telemetry:
service: "inventory"
tracing:
sampler: "always_off"
exporters:
- type: zipkin
compression: gzip
processors:
- type: batch
max-queue-size: 50
Maximum flexibility
The user configures possibly multiple processors and possibly multiple named exporters. Each processor’s configuration lists the names of the exporters it should use; no names means all exporters.
The first processor (type batch) uses both exporters because it does not
specify any exporter names. The second processor uses only the alternate-otlp
exporter.|
telemetry:
service: "inventory"
tracing:
sampler: "always_off"
exporters:
- type: zipkin
compression: gzip
name: "compressed-zipkin"
- endpoint: "http://collect.com:4317"
name: "alternate-otlp""
processors:
- type: batch
max-queue-size: 50
- type: simple
exporters: ["alternate-otlp"]
Tracing Configuration
The settings under signals.tracing prepare an OpenTelemetry TracerProvider.
When your application uses the Helidon tracing API to obtain a Tracer, Helidon
uses the TracerProvider prepared from this config to create the tracer.
The next table describes the OpenTelemetry tracing settings.
Configuration options
| Key | Type | Description |
|---|---|---|
span- | Span | Tracing span limits |
attributes | Typed | Name/value pairs passed to OpenTelemetry |
processors | List< | Settings for span processors |
exporters | Map< | Span exporters |
sampler | Sampler | Tracing sampler |
OpenTelemetry applies the defaults described in the next table.
| Setting | OpenTelemetry default (and OpenTelemetry doc link) |
|---|---|
exporters | otlp with grpc protocol - see "Properties: exporters, otel.traces.exporter property" |
processors | batch with defaults - see "Properties for batch span processor(s)" |
sampler | parentbased_always_on - see "Properties for sampler" |
span-limits | See tracing "Properties for span limits" |
Span Sampler
OpenTelemetry offers different ways of sampling data—deciding which tracing spans tp capture and send to the backend. The OpenTelemetry documentation describes sampling in more detail.
Helidon configuration supports the sampler implementations that reside in the
opentelemetry-sdk as listed in the table below. Other samplers are in other
components. If you need to use one of those:
- Add the relevant OpenTelemetry dependency to your project.
- Instantiate the span sample you need.
- Prepare the sampler and the OpenTelemetry-related builders programmatically
and use your sampler to assign the sampler the
OpenTelemetryTracer.Buildershould use.
Configuration options
| Key | Type | Default | Description |
|---|---|---|---|
param | Double | Sampler parameter | |
type | Sampler | DEFAULT | Sampler type |
Span Limits
OpenTelemetry allows you to constrain certain aspects of the data it gathers in tracing spans. By assigning the settings in the table below, you can apply the span limits you want.
Configuration options
| Key | Type | Description |
|---|---|---|
max- | Integer | Maximum attribute value length |
max- | Integer | Maximum number of events |
max- | Integer | Maximum number of attributes per event |
max- | Integer | Maximum number of attributes |
max- | Integer | Maximum number of attributes per link |
max- | Integer | Maximum number of links |
The OpenTelemetry documentation describes the defaults; see the "Properties for span limits" section there.
OpenTelemetry defaults for span limits:
| Setting | OpenTelemetry Default |
|---|---|
max-attribute-value-length | no limit |
max-attributes | 128 |
max-attributes-per-event | 128 |
max-events | 128 |
max-links | 128 |
Metrics Configuration
The settings under signals.metrics prepare an OpenTelemetry MeterProvider.
If your code uses the OpenTelemetry API to obtain an OpenTelemetry meter, meter
provider, or meter builder, OpenTelemetry uses the MeterProvider prepared from
this configuration.
The sections below describe Helidon config settings that correspond very directly to OpenTelemetry builders for the relevant OpenTelemetry type. Refer to the relevant OpenTelemetry documentation or Javadoc to understand the effect each setting has.
Configuration options
| Key | Type | Description |
|---|---|---|
readers | List< | Settings for metric readers |
attributes | Typed | Name/value pairs passed to OpenTelemetry |
views | List< | Metric view information, configurable using io. |
exporters | Map< | Metric exporter configurations, configurable using io. |
OpenTelemetry applies the defaults described in the next table.
| Setting | OpenTelemetry default (and OpenTelemetry doc link) |
|---|---|
exporters | otlp with grpc protocol - see that web page’s "Properties: exporters, otel.metrics.exporter property"] section. |
readers | PeriodicMetricReader with an interval of one minute |
The following example illustrates some of the ways you can configure OpenTelemetry metrics behavior. It is neither complete nor typical.
Metric Exporters
The configuration for metrics exporters has several additional settings beyond those described earlier for exporters in general.
Configuration options
| Key | Type | Default | Description |
|---|---|---|---|
headers | Map< | Headers added to each export message | |
memory- | Memory | Memory mode | |
temporality- | Metric | Preferred output aggregation technique (how transmitted values reflect the values recorded locally), configurable as a io. value: CUMULATIVE, | |
internal- | Internal | Self-monitoring telemetry OpenTelemetry should collect | |
certificate | Resource | Trusted certificates | |
type | Metric | OTLP | Metric exporter type |
timeout | Duration | Exporter timeout | |
connect- | Duration | Connection timeout | |
endpoint | URI | Endpoint of the collector to which the exporter should transmit | |
protocol | String | DEFAULT | Exporter protocol type |
client. | Resource | TLS client key | |
default- | Default | Preferred default histogram aggregation technique, configurable as io. | |
client. | Resource | TLS certificate | |
compression | Compression | Compression the exporter uses | |
retry- | Retry | Retry policy |
Metric Aggregation
OpenTelemetry allows control over how each exporter aggregates histogram data prior to transmission to a backend.
Configuration options
| Key | Type | Description |
|---|---|---|
type | Metric | Type of aggregation default |
You can configure the explicit bucket boundaries for EXPLICIT_BUCKET_HISTOGRAM
aggregation.
| Key | Type | Description |
|---|---|---|
bucket- | List< | Explicit bucket boundaries |
You can configure the exponential histogram aggregation behavior.
| Key | Type | Description |
|---|---|---|
max- | Integer | Maximum number of buckets |
max- | Integer | Maximum scale |
Metric Readers
An OpenTelemetry metric reader collects metric data in the server and then uses the associated metric exporter to send that data to the endpoint configured.
Configuration options
| Key | Type | Default | Description |
|---|---|---|---|
exporter | String | Name of the configured metric exporter to use for this metric reader | |
type | Metric | PERIODIC | Metric reader type |
The periodic reader supports the following settings.
| Key | Type | Default | Description |
|---|---|---|---|
exporter | String | Name of the configured metric exporter to use for this metric reader | |
interval | Duration | Metric reader read interval | |
type | Metric | PERIODIC | Metric reader type |
Metric Views
OpenTelemetry metric views allow you to influence how meters are aggregated for reporting to backend systems.
Configuration options
| Key | Type | Description |
|---|---|---|
cardinality- | Integer | Cardinality limit |
instrument- | Instrument | Instrument selector, configurable using io. |
attribute- | Predicate | Attribute name filter, configurable as a string compiled as a regular expression using java. |
name | String | Metrics view name |
description | String | Metric view description |
aggregation | Aggregation | Aggregation for the metric view, configurable as an io.: DROP, |
The instrument selector controls which meters this view reflects.
| Key | Type | Description |
|---|---|---|
unit | String | Instrument unit |
meter- | String | Meter name |
name | String | Instrument name |
meter- | String | Meter schema URL |
meter- | String | Meter version |
type | Instrument | Instrument type |
Logging Configuration
The settings under signal.logging prepare an OpenTelemetry LoggerProvider.
The sections below describe Helidon config settings that correspond directly to OpenTelemetry builders for the relevant OpenTelemetry type. Refer to the relevant OpenTelemetry documentation or Javadoc to understand the effect each setting has.
Configuration options
| Key | Type | Description |
|---|---|---|
trace- | Boolean | Whether to include only log records from traces which are sampled |
attributes | Typed | Name/value pairs passed to OpenTelemetry |
minimum- | Severity | Minimum severity level of log records to process |
processors | List< | Settings for logging processors |
enabled | Boolean | Whether the OpenTelemetry logger should be enabled |
log- | Log | Log limits to apply to log transmission |
exporters | Map< | Log record exporters |
OpenTelemetry uses the following defaults:
| Setting | OpenTelemetry default |
|---|---|
exporters | none |
minimum-severity | undefined severity number |
processors | no-op processor |
trace-based | false |
Log Limits
For defaults, Helidon defers to the OpenTelemetry defaults, listed below.
Configuration options
| Key | Type | Description |
|---|---|---|
max- | Integer | Maximum length of an attribute value |
max- | Integer | Maximum number of attributes allowed |
OpenTelemetry applies the following defaults:
| Setting | OpenTelemetry default |
|---|---|
max-attribute-value-length | Integer.MAX_VALUE |
max-number-of-attributes | 128 |
exporters | otlp with grpc protocol - see that web page’s "Properties: exporters, otel.logger.exporter property" section. |
log-limits | max- |
processors | otlp - see that web page’s "Properties: logs" section. |
The following example illustrates some of the ways you can configure OpenTelemetry logger behavior. It is neither complete nor typical.