Metrics

Overview

Helidon metrics is a neutral metrics API which provides

• a unified way for Helidon servers to export monitoring data—telemetry—to management agents, and
• a unified Java API which all application programmers can use to register and update meters to expose telemetry data from their services.

Metrics is one of the Helidon observability features.

Terminology

Helidon uses the term "metrics" to refer to the subsystem in Helidon which manages the registration of, updates to, and reporting of aggregate statistical measurements about the service. The term "meter" refers to an entity which collects these measurements, such as a counter or a timer.

Maven Coordinates

To enable metrics, add the following dependency to your project’s pom.xml (see Managing Dependencies).

<dependency>
  <groupId>io.helidon.metrics</groupId>
  <artifactId>helidon-metrics-api</artifactId>
</dependency>

This dependency adds the metrics API and a no-op implementation of that API to your project. The no-op implementation:

• does not register meters in a registry
• does not update meter values
• does not expose the metrics endpoint for reporting meter values.

To include the full-featured metrics implementation and support for the metrics endpoint, add the following dependency to your project:

Packaging the metrics endpoint support and a full-featured metrics implementation:

<dependency>
  <groupId>io.helidon.webserver.observe</groupId>
  <artifactId>helidon-webserver-observe-metrics</artifactId>
</dependency>

Adding this dependency packages the full-featured metrics implementation and support for the metrics endpoint with your service.

You might notice the transitive dependency io.helidon.metrics.providers:helidon-metrics-providers-micrometer in your project. This component contains an implementation of the Helidon metrics API that uses Micrometer as the underlying metrics technology.

Helidon provides several built-in meters in a separate artifact. To include the build-in meters, add the following dependency to your project:

Packaging the built-in meters:

<dependency>
  <groupId>io.helidon.metrics</groupId>
  <artifactId>helidon-metrics-system-meters</artifactId>
  <scope>runtime</scope>
</dependency>

Instrumenting Your Service

You add meters to your service by writing code which explicitly invokes the metrics API to register meters, retrieve previously-registered meters, and update meter values.

Later sections of this document describe how to do this.

Meter Types

Helidon supports meters inspired by Micrometer and summarized in the following table:

Types of Meters

Meter Registry

Helidon stores all meters in a meter registry. Typically, applications use the global meter registry which is the registry where Helidon stores built-in meters. Application code obtains the global registry by injecting MeterRegistry or, for imperative code, using Services.get(MeterRegistry.class).

Publishing Metrics

Helidon’s Micrometer-based metrics implementation includes these ways of publishing metrics data to external systems:

• Prometheus/OpenMetrics
• OTLP (OpenTelemetry Protocol)

You can configure publishers in the publishers configuration section under the top level metrics node or under server.features.observe.observers.metrics. If you do not set up publishers explicitly, Helidon uses an inferred Prometheus publisher for backward compatibility. See this later section for details.

Publishers in Helidon’s Micrometer-based metrics implementation use Micrometer MeterRegistry implementations. Each Helidon meter registry owns a composite Micrometer registry containing one registry for each enabled publisher. This has these important effects:

• Meters which Helidon or your code registers in a Helidon meter registry are registered in all active publisher registries owned by that registry.
• Each Helidon meter has an implementation in every active publisher registry belonging to its Helidon registry.
• When Helidon or your code updates a Helidon meter, Micrometer applies the change to every corresponding meter in the publisher registries belonging to that Helidon registry.

As a result, configuring more than one active publisher for a Helidon meter registry can affect performance.

OpenTelemetry Protocol

If you configure an OTLP publisher, Helidon exports metrics data periodically to a backend system you configure.

Configuration options

The configuration directly mirrors the Micrometer OtlpMeterRegistry settings so you can control all behavior which Micrometer exposes for the meter registry.

The following example sets up an OTLP publisher to transmit metrics data every 30 seconds.

Example OTLP publisher settings:

metrics:
  publishers:1
    otlp:2
      interval: PT30S
      url: 'http://somehost.com:4318/v1/metrics'

Prometheus Publisher

If you configure a Prometheus publisher or rely on the inferred one, Helidon can make the metrics data available in the Prometheus/OpenMetrics format. (To serve the data at the metrics endpoint in your service, your project must also depend on the Helidon metrics observer component.)

Configuration options

Inferred Publisher

As described earlier, Helidon prepares an inferred Prometheus publisher if you do not set up any publishers.

Note that Helidon uses the inferred publisher only if you add no publishers explicitly, either in the configuration or programmatically. If you specify any publishers explicitly, Helidon uses only the ones you set up.

In particular, Helidon does not use the inferred Prometheus publisher if you create a metrics.publishers section containing only an OTLP publisher.

You can configure other publishers and still have Helidon use the default one by simply adding the prometheus publisher entry. You do not need to specify further settings for it.

Using an OTLP publisher and the default Prometheus publisher:

metrics:
  publishers:
    prometheus:
    otlp:
      interval: PT20S

Metrics Endpoint

When you add the helidon-webserver-observe-metrics dependency to your project, Helidon provides a built-in REST endpoint /observe/metrics which responds with a report of the registered meters and their values.

Clients can request a particular output format from the endpoint.

Formats for /observe/metrics output:

Clients can narrow the report to a specific meter name using the name query parameter, such as /observe/metrics?name=myCount.

Example Reporting: Prometheus format:

curl -s -H 'Accept: text/plain' -X GET http://localhost:8080/observe/metrics

# HELP classloader_loadedClasses_count Displays the number of classes that are currently loaded in the Java virtual machine.
# TYPE classloader_loadedClasses_count gauge
classloader_loadedClasses_count 5297.0

See the summary of the OpenMetrics and Prometheus Format for more information.

Example Reporting: JSON format:

curl -s -H 'Accept: application/json' -X GET http://localhost:8080/observe/metrics

{
  "memory.maxHeap" : 3817865216,
  "memory.committedHeap" : 335544320
}

In addition to your application meters, the reports contain other meters of interest such as system and VM information.

Format

The OpenMetrics format and the Prometheus exposition format are very similar in most important respects but are not identical. This brief summary treats them as the same.

The OpenMetrics/Prometheus format represents each meter using three lines of output as summarized in the following table.

The OpenMetrics/Prometheus output converts meter IDs in these ways:

• Names in camel case are converted to "snake case" and dots are converted to underscores.
• Names include any units specified for the meter.
• For percentiles, the ID includes a tag identifying which percentile the line of output describes.

As the earlier example output showed, for a meter with multiple values, the OpenMetrics/Prometheus output reports a "metric family" which includes a separate family member meter for each of the multiple values.

The name for each member in the family is derived from the registered name for the meter plus a suffix indicating which one of the meter’s multiple values the line refers to.

The following table summarizes the naming for each meter type:

¹ The OpenMetrics/Prometheus output format reports a timer as a summary with units of seconds.

JSON Format

Unlike OpenMetrics/Prometheus output, which combines the data and the metadata in a single response, you use an HTTP GET request to retrieve metrics JSON data and an OPTIONS request to retrieve metadata in JSON format.

Helidon reports meters directly in a single JSON object.

JSON metrics metadata output (partial):

{
  "getTimer": {
    "type": "timer",
    "unit": "seconds",
    "description": "Timer for getting the default greeting"
  },
  "requests.count": {
    "type": "counter",
    "description": "Each request (regardless of HTTP method) will increase this counter"
  },
  "cpu.systemLoadAverage": {
    "type": "gauge",
    "description": "Displays the system load average for the last minute."
  },
  "classloader.loadedClasses.count": {
    "type": "gauge",
    "description": "Displays the number of classes that are currently loaded in the Java virtual machine."
  }
}

Understanding the JSON Metrics Data Format

The Helidon JSON format expresses each meter as either a single value (for example, a counter) or a structure with multiple values (for example, a timer).

JSON output for a single-valued meter (for example, Counter):

"requests.count": 5

JSON output for a multivalued meter (for example, Timer):

"getTimer": {
  "count": 3,
  "max": 0.0030455,
  "mean": 0.0011060836666666666,
  "elapsedTime": 0.003318251,
  "p0.5": 0.000151552,
  "p0.75": 0.003141632,
  "p0.95": 0.003141632,
  "p0.98": 0.003141632,
  "p0.99": 0.003141632,
  "p0.999": 0.003141632
}

By default, Helidon formats time values contained in JSON output as seconds. You can change this behavior as described below.

Understanding the JSON Metrics Metadata Format

Access the metrics endpoint with an HTTP OPTIONS request and the Accept: application/json header to retrieve metadata in JSON format.

Example Counter metadata:

"requests.count": {
  "type": "counter",
  "description": "Each request (regardless of HTTP method) will increase this counter"
    }

Example Timer metadata:

"getTimer": {
  "type": "timer",
  "unit": "seconds",
  "description": "Timer for getting the default greeting"
}

Generally, the output for a given meter reflects only the metadata that the application or Helidon code explicitly set on that meter.

One exception is that metadata for a timer always includes the unit field. By default, Helidon formats timer data in JSON output as seconds, regardless of any explicit baseUnit setting applied to the timers. But as described below you can change this behavior which can lead to different timers being formatted using different units. Checking the metadata is the only way to know for sure what units Helidon used to express a given timer, so Helidon always includes unit in timer metadata.

Controlling JSON Timer Output

By default, Helidon expresses timer data as seconds.

You can change this using configuration:

Setting default timer units for JSON in application.yaml:

metrics:
  timers:
    json-units-default: units

• For units specify any valid name for a TimeUnit value (SECONDS, MILLISECONDS, etc.)

If you have configured json-units-default, Helidon formats each timer’s data as follows:

1. If code set baseUnit on the timer, Helidon uses those units for that timer.
2. Otherwise, Helidon uses the default units you configured.

To enable the JSON output behavior from Helidon 3, specify json-units-default as NANOSECONDS.

Enabling the Metrics REST Service

If you add the dependencies described above, your service automatically supports the metrics REST endpoint as long as the WebServer is configured to discover features automatically.

If you disable auto-discovery, you can add the metrics observer explicitly.

1. Create an instance of MetricsObserver, either directly as shown below or using its builder.
2. Include the MetricsObserver instance in your application’s ObserveFeature.
3. Register your ObserveFeature with your WebServer.

ObserveFeature observe = ObserveFeature.builder()
        .config(config.get("server.features.observe"))
        .addObserver(MetricsObserver.create())
        .build();

WebServer server = WebServer.builder()
        .config(Config.global().get("server"))
        .featuresDiscoverServices(false)
        .addFeature(observe)
        .routing(Main::routing)
        .build()
        .start();

API

To work with Helidon Metrics in your code, follow these steps:

1. Use injection or Services.get(MeterRegistry.class) to get a reference to the global MeterRegistry instance.
2. Use the MeterRegistry instance and builders from its MetricsFactory to register new meters and look up previously registered meters.
3. Use the meter reference returned from the MeterRegistry to update the meter or get its value.

You can also use the MeterRegistry to remove an existing meter.

Helidon 27 removes the former static convenience methods from Metrics. Use these replacements instead:

• Replace Metrics.globalRegistry() with injection of MeterRegistry or Services.get(MeterRegistry.class).
• Replace Metrics.createMeterRegistry() with Services.get(MetricsFactory.class).createMeterRegistry(MetricsConfig.create()).
• Replace Metrics.createMeterRegistry(metricsConfig) with Services.get(MetricsFactory.class).createMeterRegistry(metricsConfig).
• Replace Metrics.getOrCreate(builder) with meterRegistry.getOrCreate(builder).
• Replace the Metrics.getCounter, Metrics.getSummary, Metrics.getGauge, Metrics.getTimer, and Metrics.get lookup helpers with the corresponding MeterRegistry lookup methods.
• Replace Metrics.tag(key, value) with Services.get(MetricsFactory.class).tagCreate(key, value), and create multiple tags using the same MetricsFactory instance.

Helidon Metrics API

The Helidon Metrics API defines the classes and interfaces for meter types and other related items.

The following table summarizes the meter types.

Meter Types

Each meter type has its own set of methods for updating and retrieving the value.

MeterRegistry

To register or look up meters programmatically, your service code uses the global MeterRegistry. Inject MeterRegistry into your service or invoke Services.get(MeterRegistry.class) to get a reference to it.

To locate an existing meter or register a new one, your code:

1. Obtains the registry-owned MetricsFactory using meterRegistry.metricsFactory() and creates a builder of the appropriate meter type, setting the name and possibly other characteristics.
2. Invokes the MeterRegistry.getOrCreate method, passing the builder.

The meter registry returns a reference to a previously-registered meter with the specified name and tags or, if none exists, a newly-registered meter. Your code can then operate on the returned meter as needed to record new measurements or retrieve existing data.

The example code in the Examples section below illustrates how to register, retrieve, and update meters.

Understanding Timers, Units, and Output

Your application can assign the meter builder’s Meter.Builder baseUnit setting for any meter your application creates. In particular, the Timer.Builder baseUnit method allows code to assign a baseUnit for a timer, passing a Java TimeUnit value. The timer builder also has the String variant of the baseUnit method and enforces that the value corresponds (case-insensitively) to one of the TimeUnit enum values.

Note that, regardless of the baseUnit setting for a Timer, by convention and specification Prometheus output expresses time values in seconds.

By default, the same is true of Helidon’s JSON format: timer values are displayed in seconds regardless of any timer’s baseUnit setting. You can override this as described in the Controlling Timer Output section, in which case the JSON output for each timer reflects its baseUnit setting.

Accessing the Underlying Implementation: unwrap

The neutral Helidon metrics API is an abstraction of common metrics behavior independent of any given implementation. As such, we intentionally excluded some implementation-specific behavior from the API.

Sometimes you might want access to methods that are present in a particular metrics implementation but not in the Helidon API. Helidon allows that via the unwrap method on the meter types and on their builders. Each full implementation of the Helidon meter types and their builders refers to a delegate meter or delegate builder internally. The unwrap method lets you obtain the delegate, cast to the type you want.

Of course, using this technique binds your code to a particular metrics implementation.

The Wrapper interface declares the unwrap method which accepts a class parameter to which the delegate is cast. You can then invoke any method declared on the implementation-specific type.

Configuration options

Metrics Configuration Migration

Metrics Scopes

Metrics scopes were a MicroProfile-specific feature and are no longer part of the Helidon core metrics model in Helidon 27. The scope APIs remain temporarily for compatibility, are deprecated for removal, and have no effect. In particular:

• meter and meter builder scope methods return no scope or ignore the provided scope;
• metrics.scoping configuration can still be parsed but does not affect meters or output;
• the scope query parameter is ignored; and
• the legacy /observe/metrics/application, /observe/metrics/base, and /observe/metrics/vendor paths expose the same unscoped metrics as /observe/metrics.

An integration which needs equivalent classification can implement it using ordinary meter tags:

1. Choose an integration-owned tag name and values.
2. Provide a MeterBuilderCustomizer service. Override customize(Meter.Builder) and inspect Meter.Builder.origin() to select a tag value based on the fully-qualified name of the type which originated a meter. Helidon supplies the provider class name automatically for meters contributed by a MetersProvider.
3. Create the tag using MetricsFactory.tagCreate and add it to the builder before registration. Other meter producers can provide origin information using Meter.Builder.origin(String) and then register the builder using MeterRegistry.getOrCreate(builder). Do not use the deprecated scope methods.
4. Translate any integration-specific selection into a tag-selection map and pass it to the provider-neutral MeterRegistryFormatterProvider.formatter overload. The integration owns its routes, query parameters, and response semantics.

Helidon 27 always reports the system-provided gc.time meter as a Gauge. The former metrics.gc-time-type compatibility setting is no longer supported; remove it from configuration. Application code that looked up gc.time as a Counter should use the MeterRegistry gauge lookup APIs instead.

The deprecated metrics.rest-request-enabled compatibility setting is also no longer supported in Helidon 27. Replace it with metrics.rest-request.enabled.

Helidon 27 uses Prometheus Java Client 1.7.0 through Micrometer instead of the legacy Prometheus simpleclient integration. By default, metric and tag names use the new client's normalization. In particular, names which do not begin with a letter are normalized using the new client's rules, and reserved suffixes such as _total, _created, _bucket, and _info are removed from base names so the writer can add type-appropriate suffixes.

To retain the Prometheus names emitted by earlier Helidon releases for counters, functional counters, gauges, timers, and distribution summaries, configure the legacy non-letter prefix:

metrics:
  publishers:
    prometheus:
      naming-convention:
        non-letter-prefix: "m_"

Setting non-letter-prefix selects legacy normalization as a whole, including preserving user-supplied reserved suffixes. The value must match [A-Za-z][A-Za-z0-9_]*; m_ reproduces the earlier Helidon prefix, while another valid value uses the same legacy rules with that prefix. The naming-convention.timer-suffix setting remains available in both modes. The publisher's existing prefix setting controls Micrometer property lookup and does not prefix metric names.

The former metrics.prometheus.histogramFlavor property is accepted so existing configuration continues to load, but the new integration ignores it and logs a warning.

Code which accesses Prometheus-specific Helidon APIs or implements the exemplar SPI needs these type changes:

• PrometheusPublisher.prometheusRegistry() now supplies io.micrometer.prometheusmetrics.PrometheusMeterRegistry instead of io.micrometer.prometheus.PrometheusMeterRegistry.
• SpanContextSupplierProvider.get() now returns io.prometheus.metrics.tracer.common.SpanContext instead of io.prometheus.client.exemplars.tracer.common.SpanContextSupplier; the new type reports trace ID, span ID, sampled state, and when the current span is selected as an exemplar.

Metrics Observer

Helidon can make the registered meters and their current values available externally at an endpoint (/observe/metrics by default). You can control aspects of how Helidon furnishes this information under the server.features.observe.observers.metrics configuration section.

See the Helidon OpenTelemetry documentation for more information.

Selecting REST Endpoints for Automatic Measurement

You can choose which endpoints to include in Helidon’s automatic measurements using the auto-http-metrics config section.

Configuration options

Deprecated Options

The paths section contains zero or more entries, each entry having the following settings:

Helidon decides whether to measure incoming requests as follows:

• If you omit the auto-http-metrics configuration, Helidon measures all endpoints.
• If you specify the auto-http-metrics configuration, by default Helidon does not measure built-in endpoints such as metrics, health, and openapi. You can add items under auto-http-metrics.paths to control more exactly which endpoints to measure.
• If you include the paths section, Helidon checks a request against the path entries in order. A given request matches an entry if its path matches the path pattern and its HTTP method is in the methods list. If there is no methods list for an entry, all HTTP methods match the entry.
• Method names configured in path entries are matched exactly.
• If a request matches an entry, the entry’s enabled setting determines if the request should be measured.
• If a request matches multiple entries, the first match wins.
• If a request matches no entry, it is measured.

The auto-http-metrics.sockets setting controls which sockets are included in the measurements; if not set, Helidon measures requests on all sockets.

Controlling HTTP Method Attribute Values

For the OpenTelemetry http.server.request.duration metric, the auto-http-metrics.known-methods setting controls the values Helidon records for the http.request.method attribute. This setting does not control which requests Helidon measures; use the methods setting in a paths entry for that purpose.

By default, Helidon records CONNECT, DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT, QUERY, and TRACE using those exact method names. Helidon records any other method, including case variants, as _OTHER. Method names in the configuration and incoming requests must match exactly, and the configured case is preserved in the recorded attribute.

Assigning known-methods replaces the entire default list. Include every default method you want to retain.

Configuring HTTP Method Attribute Values:

server:
  features:
    observe:
      observers:
        metrics:
          auto-http-metrics:
            known-methods: ["GET", "HEAD", "POST", "PROPFIND"]

With this configuration, Helidon records the four listed methods using their configured names and records every other HTTP method, including default methods omitted from the list, as _OTHER.

Including and Excluding Endpoints from Automatic Measurement:

server:
  features:
    observe:
      observers:
        metrics:
          auto-http-metrics:
            paths:
              - path: "/greet"1
                methods: ["GET","HEAD"]
              - path: "/greet/{name}"2
                enabled: false
            sockets: ["@default","private"]3

The AutoHttpMetricsConfig documentation describes the configuration more fully.

Examples

Helidon includes several pre-written example applications illustrating aspects of metrics:

• Filtering reported meters by name using the metrics endpoint
• Controlling key performance indicator metrics using configuration and KeyPerformanceIndicatorMetricsSettings.

Custom Meter

The following example, based on the Helidon QuickStart application, shows how to register and update a new Counter in application code. The counter tracks the number of times any of the service endpoints is accessed.

Define and use a Counter:

public class GreetService implements HttpService {

    private final Counter accessCtr;

    GreetService() {
        var meterRegistry = Services.get(MeterRegistry.class);1
        var metricsFactory = meterRegistry.metricsFactory();

        this.accessCtr = meterRegistry
                .getOrCreate(metricsFactory.counterBuilder("accessctr"));2
    }

    @Override
    public void routing(HttpRules rules) {
        rules
                .any(this::countAccess)3
                .get("/", this::getDefaultMessageHandler)
                .get("/{name}", this::getMessageHandler)
                .put("/greeting", this::updateGreetingHandler);

    }

    void countAccess(ServerRequest request,
                     ServerResponse response) {

        accessCtr.increment();4
        response.next();
    }

    void getDefaultMessageHandler(ServerRequest request,
                                  ServerResponse response) {
        // ...
    }

    void getMessageHandler(ServerRequest request,
                           ServerResponse response) {
        // ...
    }

    void updateGreetingHandler(ServerRequest request,
                               ServerResponse response) {
        // ...
    }
}

Perform the following steps to see the new counter in action.

Build and run the application:

mvn package
java -jar target/helidon-quickstart-se.jar

Retrieve the new counter:

curl 'http://localhost:8080/observe/metrics?name=accessctr'1

# HELP accessctr_total
# TYPE accessctr_total counter
accessctr_total 0.02

Access a service endpoint to retrieve a greeting:

curl http://localhost:8080/greet

{"message":"Hello World"}

Retrieve the counter again:

curl 'http://localhost:8080/observe/metrics?name=accessctr'

# HELP accessctr_total
# TYPE accessctr_total counter
accessctr_total 1.0 # <1>

Configuration Examples

Disabling metrics entirely:

server:
  features:
    observe:
      observers:
        metrics:
          enabled: false

Helidon does not update metrics, and the /observe/metrics endpoints respond with 404.

Virtual Threads Meters

Gathering data to compute the meters for virtual threads is designed to be as efficient as possible, but doing so still imposes a load on the server and by default Helidon does not report meters related to virtual threads.

Enabling virtual thread meters:

metrics:
  virtual-threads:
    enabled: true

Pinned Virtual Threads

Helidon measures pinned virtual threads only when the thread is pinned for a length of time at or above a threshold. Control the threshold as shown in the example below.

Setting virtual thread pinning threshold to 100 ms:

metrics:
  virtual-threads:
    pinned:
      threshold: PT0.100S

The threshold value is a Duration string, such as PT0.100S for 100 milliseconds.

Key Performance Indicator (KPI) Meters

Any time you include the Helidon metrics module in your application, Helidon tracks a basic performance indicator meter: a Counter of all requests received (requests.count)

Helidon also includes additional, extended KPI meters which are disabled by default:

• current number of requests in-flight - a Gauge (requests.inFlight) of requests currently being processed
• long-running requests - a Counter (requests.longRunning) measuring the total number of requests which take at least a given amount of time to complete; configurable, defaults to 10000 milliseconds (10 seconds)
• load - a Counter (requests.load) measuring the number of requests worked on (as opposed to received)
• deferred - a Gauge (requests.deferred) measuring delayed request processing (work on a request was delayed after Helidon received the request)

The names above use the default CAMEL built-in meter name format. With the SNAKE format, Helidon reports the in-flight and long-running metrics as requests.in_flight and requests.long_running, respectively.

You can enable and control these meters using configuration:

Controlling extended KPI meters:

server:
  features:
    observe:
      observers:
        metrics:
          key-performance-indicators:
            extended: true
            long-running:
              threshold-ms: 2000

Reference

• Micrometer Metrics concepts documentation
• OpenMetrics format
• Prometheus exposition format